Directory Structure:

└── ./
    └── nebari-docs
        └── docs
            └── docs
                ├── explanations
                │   ├── advanced-configuration.md
                │   ├── advanced-custom-settings.md
                │   ├── advanced-customize-themes.md
                │   ├── advanced-env-configuration.md
                │   ├── advanced-profiles-settings.md
                │   ├── advanced-provider-configuration.md
                │   ├── advanced-security-configuration.md
                │   ├── architecture.mdx
                │   ├── configuration-best-practices.mdx
                │   └── index.mdx
                ├── get-started
                │   ├── cloud-providers.mdx
                │   ├── deploy.mdx
                │   ├── index.mdx
                │   ├── installing-nebari.md
                │   └── quickstart.md
                ├── how-tos
                │   ├── access-logs-loki.md
                │   ├── configure-keycloak-howto.md
                │   ├── configure-smtp.md
                │   ├── connect-via-ssh.mdx
                │   ├── debug-nebari.mdx
                │   ├── develop-local-packages.md
                │   ├── domain-registry.md
                │   ├── fine-grained-permissions.md
                │   ├── index.mdx
                │   ├── install-pip-packages.md
                │   ├── jhub-app-launcher.md
                │   ├── manual-backup.md
                │   ├── nebari-aws.md
                │   ├── nebari-azure.md
                │   ├── nebari-destroy.md
                │   ├── nebari-environment-management.md
                │   ├── nebari-extension-system.md
                │   ├── nebari-gcp.md
                │   ├── nebari-idle-culler.md
                │   ├── nebari-kubernetes.mdx
                │   ├── nebari-local.md
                │   ├── nebari-stages-directory.md
                │   ├── nebari-upgrade.md
                │   ├── setup-argo.md
                │   ├── setup-healthcheck.md
                │   ├── setup-monitoring.md
                │   ├── telemetry.md
                │   ├── upgrade-kubernetes-version.md
                │   ├── use-gpus.mdx
                │   ├── using-argo.md
                │   └── using-vscode.md
                ├── references
                │   ├── container-sources.md
                │   ├── enhanced-security.md
                │   ├── index.mdx
                │   ├── personas.md
                │   └── RELEASE.md
                ├── tutorials
                │   ├── argo-workflows-walkthrough.md
                │   ├── create-dashboard.md
                │   ├── creating-new-environments.md
                │   ├── index.mdx
                │   ├── login-with-keycloak.md
                │   ├── run-notebook-as-a-job.md
                │   └── using_dask.md
                ├── faq.md
                ├── glossary.md
                ├── troubleshooting.mdx
                └── welcome.mdx



---
File: /nebari-docs/docs/docs/explanations/advanced-configuration.md
---

---
title: Advanced configuration
id: advanced-configuration
description: An in-depth guide to advanced configuration options.
---

# Advanced configuration guide

Nebari is a highly configurable tool with different customization options.
To better understand how to use these options, this guide will walk you through the different configuration options in `nebari-config.yaml` and how to use them.
In the "How to deploy Nebari" pages of our docs we covered how you can auto-generate this file using `nebari init` (and properly set options/flags and environment variables).

After first initializing a project, you can find the configuration file, `nebari-config.yaml`, in your project directory.
This file is a `YAML` file that exports sets of parameters used by Nebari to deploy and redeploy changes to your infrastructure.

<details>
<summary>Complete configuration example.</summary>

The different sections of the config will be covered in more detail below.

```yaml
nebari_version: 2023.7.2

project_name: demo
namespace: dev
provider: gcp
domain: demo.nebari.dev

ci_cd:
  type: gitlab-ci
  branch: main
  commit_render: true
  before_script:
    - echo "running commands before ci completes"
  after_script:
    - echo "running commands after ci completes"
    - echo "additional commands to run"

certificate:
  type: lets-encrypt
  acme_email: dev@nebari.dev
  acme_server: https://acme-v02.api.letsencrypt.org/directory

security:
  authentication:
    type: Auth0
    config:
      client_id: cLiEnT123Id456
      client_secret: cClIeNt789123sEcReT4567890
      auth0_subdomain: qhub-dev
  keycloak:
    initial_root_password: 1n1t1alr00tp@ssw0rd

default_images:
  jupyterhub: quay.io/nebari/nebari-jupyterhub:2023.7.2
  jupyterlab: quay.io/nebari/nebari-jupyterlab:2023.7.2
  dask_worker: quay.io/nebari/nebari-dask-worker:2023.7.2

storage:
  conda_store: 500Gi
  shared_filesystem: 500Gi

theme:
  jupyterhub:
    hub_title: My Nebari Platform
    hub_subtitle: Your open source data science platform, hosted on Google Cloud Platform
    welcome: Welcome! Learn about Nebari's features and configurations in <a href="https://www.nebari.dev/docs">the
      documentation</a>. If you have any questions or feedback, reach the team on
      <a href="https://www.nebari.dev/docs/community#getting-support">Nebari's support
      forums</a>!!
    logo: https://raw.githubusercontent.com/nebari-dev/nebari-design/main/logo-mark/horizontal/Nebari-Logo-Horizontal-Lockup-White-text.svg
    display_version: true

jupyterlab:
  idle_culler:
    terminal_cull_inactive_timeout: 30
    kernel_cull_idle_timeout: 30
    server_shutdown_no_activity_timeout: 30

helm_extensions: []
monitoring:
  enabled: true
argo_workflows:
  enabled: true
kbatch:
  enabled: true

terraform_state:
  type: remote

google_cloud_platform:
  project: gcp_project_id
  region: us-central1
  kubernetes_version: 1.26.7-gke.500
  tags:
  - "my-custom-tags"

  node_groups:
    general:
      instance: n1-standard-8
      min_nodes: 1
      max_nodes: 1

    user:
      instance: n1-standard-4
      min_nodes: 0
      max_nodes: 200

    worker:
      instance: n1-standard-4
      min_nodes: 0
      max_nodes: 1000

    gpu-tesla-k80-x1:
      instance: "n1-standard-8"
      min_nodes: 0
      max_nodes: 50
      guest_accelerators:
        - name: nvidia-tesla-k80
          count: 1

    gpu-ampere-a100-x1:
      instance: a2-highgpu-1g
      min_nodes: 0
      max_nodes: 1

profiles:
  jupyterlab:
  - display_name: Small Instance
    description: Stable environment with 2 cpu / 8 GB RAM
    default: true
    kubespawner_override:
      cpu_limit: 2
      cpu_guarantee: 1.5
      mem_limit: 8G
      mem_guarantee: 5G

  - display_name: Medium Instance
    description: Stable environment with 4 cpu / 16 GB RAM
    kubespawner_override:
      cpu_limit: 4
      cpu_guarantee: 3
      mem_limit: 16G
      mem_guarantee: 10G

  - display_name: A100 GPU Instance 1x
    access: yaml
    groups:
      - gpu-access
    description: GPU instance with 12 cpu / 85GB RAM / 1 Nvidia A100 GPU (40 GB GPU RAM)
    kubespawner_override:
      cpu_limit: 12
      cpu_guarantee: 10
      mem_limit: 85G
      mem_guarantee: 75G
      image: quay.io/nebari/nebari-jupyterlab-gpu:2023.7.2
      extra_pod_config:
        volumes:
        - name: "dshm"
          emptyDir:
            medium: "Memory"
            sizeLimit: "2Gi"
      extra_container_config:
        volumeMounts:
        - name: "dshm"
          mountPath: "/dev/shm"
      extra_resource_limits:
        nvidia.com/gpu: 1
      node_selector:
        "cloud.google.com/gke-nodepool": "gpu-ampere-a100-x1"


  dask_worker:
    Small Worker:
      worker_cores_limit: 2
      worker_cores: 1.5
      worker_memory_limit: 8G
      worker_memory: 5G
      worker_threads: 2

    Medium Worker:
      worker_cores_limit: 4
      worker_cores: 3
      worker_memory_limit: 16G
      worker_memory: 10G
      worker_threads: 4

    GPU Worker k80:
      worker_cores_limit: 2
      worker_cores: 1.5
      worker_memory_limit: 8G
      worker_memory: 5G
      worker_threads: 2
      image: quay.io/nebari/nebari-dask-worker-gpu:2023.7.2
      worker_extra_pod_config:
        nodeSelector:
          "cloud.google.com/gke-nodepool": "gpu-tesla-k80-x1"
      worker_extra_container_config:
        resources:
          limits:
            nvidia.com/gpu: 1



environments:
  environment-dask.yaml:
    name: dask
    channels:
    - conda-forge
    dependencies:
    - python=3.10.8
    - ipykernel=6.21.0
    - ipywidgets==7.7.1
    - nebari-dask ==2023.1.1
    - python-graphviz=0.20.1
    - pyarrow=10.0.1
    - s3fs=2023.1.0
    - gcsfs=2023.1.0
    - numpy=1.23.5
    - numba=0.56.4
    - pandas=1.5.3
    - pip:
      - kbatch==0.4.2

conda_store:
  image_tag: v0.4.14
  extra_settings:
    CondaStore:
      conda_allowed_channels:
        - main
        - conda-forge
```

</details>

## General configuration settings

The `nebari-config.yaml` file can be split into several sections.

The first section is the version of Nebari you wish to run.

```yaml
### Nebari version ###
nebari_version: 2023.7.2
```

:::note
You will get a validation error if the version of `nebari` used from the command line is different from the one in the `nebari-config.yaml`.
:::

The next section relates to Nebari's inner mechanics for the initial deployment and is the most important section of the configuration file,
because the following parameters are heavily propagated throughout all infrastructure components.

```yaml
### General configuration ###
project_name: demo
namespace: dev
provider: gcp
domain: demo.nebari.dev
```

`project_name`: Determines the base name for all major infrastructure related resources on Nebari. Should be compatible with the Cloud provider's naming conventions. See [Project Naming Conventions](/docs/explanations/configuration-best-practices.mdx#naming-conventions) for more details.

`namespace`: Used in combination with `project_name` to label infrastructure related resources on Nebari and also determines the target [_namespace_](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) used when deploying kubernetes resources. Defaults to `dev`.

`provider`: Determines the cloud provider used to deploy infrastructure related resources on Nebari. Possible values are:

- `aws` for Amazon Web Services
- `gcp` for Google Could Provider
- `azure` for Microsoft Azure
- `existing` for deploying on an existing Kubernetes infrastructure
- `local` for local cluster deployment using Kind

`domain`: The top level URI used to access the application services.

<!-- For more information regarding the format of this field, see [Domain Format](/docs/explanations/config-best-practices#domain-format). -->
<!-- TODO: Complete the Domain Format section and then link to it -->

### Continuous integration and continuous deployment

Nebari uses [infrastructure-as-code](https://en.wikipedia.org/wiki/Infrastructure_as_code) to maintain a description of the deployed infrastructure in source control. By using a git repository with CI/CD configured, teams can more quickly modify their deployment, empowering developers and data scientists to request the changes and have them approved by an administrator.

When a `ci_cd` section is configured within your `nebari-config.yaml`, the first `nebari deploy` command will create all related files that describe a [CI/CD](https://about.gitlab.com/topics/ci-cd/) process. These pipelines will then be responsible for redeploying Nebari as changes are made to a specified branch. (Alternatively, an administrator can use `nebari render` to generate the necessary files as if running a dry-run.) Currently, Nebari can generate CI/CD for [GitHub Actions](https://docs.github.com/en/actions) and [GitLab CI](https://docs.gitlab.com/ee/ci/).

Below is an example `ci_cd` section in a `nebari-config.yaml` file.

```yaml
### Continuous integration ###
ci_cd:
  type: gitlab-ci # 'gitlab-ci' or 'github-actions'
  branch: main # Branch that triggers deployment
  commit_render: true # During deployment, commit the rendered IaC back into the repository
  before_script: # GitLab only
    - echo "running commands before ci completes"
  after_script: # GitLab only
    - echo "running commands after ci completes"
    - echo "additional commands to run"
```

`ci_cd` (optional): Used to enable continuous integration and continuous deployment (CI/CD) frameworks on Nebari.

- `type`: Current supported CI providers are `github-actions` and `gitlab-ci`
- `branch`: git branch on which to commit changes from `nebari render`
- `commit_render`: Whether to commit the rendered changes back into the repo. Optional, defaults to `true`.
- `before_script` (optional): Script to run before CI starts Nebari infrastructure deployment. This is useful in cases that additional setup is required for Nebari to deploy the
  resources. Currently only supported on `gitlab-ci`.
- `after_script` (optional): Script to run after CI ends infrastructure deployment. This is useful in cases to notify resources of successful Nebari deployment. Currently supported on `gitlab-ci`.

The CI/CD workflow that is best for you will depend on your organization, but the following tenets will be appropriate for most situations.

- You will want to have an upstream Git repository configured - we recommend either GitHub or GitLab since we support generating CI/CD jobs for these products.
- The branch that triggers deployment (typically `main`, but you can set other ones in Nebari config's `ci_cd.branch`) should be protected so that only sys admins can commit or approve pull (or merge) requests into it.
- CI/CD variables must be set in your repository so the pipeline can access your cloud (see Note below)
- Non-admin users who have write access to the repository's non-protected branches may create their own branch off of `main`, locally make changes to the `nebari-config.yaml` and other files, and then push that branch to the origin and propose they be deployed via a Pull Request.
- Advanced Nebari users may also want to add a step in their deployment flow that includes a `nebari render` so that the administrator may preview the resulting diffs to IaC and/or CI/CD files before `nebari deploy` is executed.

:::note
In order for your CI/CD pipeline to be able to deploy changes into your Nebari cloud hosting provider, you must set the appropriate authentication environment variables for your GitLab or GitHub CI/CD execution environment. See the Authentication section for deploing to [AWS](https://www.nebari.dev/docs/how-tos/nebari-aws/#authentication), [Azure](https://www.nebari.dev/docs/how-tos/nebari-azure#authentication), or [GCP](https://www.nebari.dev/docs/how-tos/nebari-gcp/#authentication) for Nebari's required variables. Guidance on how to set these for your repository/project can be found in the documentation for [GitHub Actions](https://docs.github.com/en/actions/learn-github-actions/variables) and [GitLab CI/CD](https://docs.gitlab.com/ee/ci/variables/).
:::

### Certificates

To enable HTTPS on your website, you need to get a SSL certificate (a type of file) from a Certificate Authority (CA).
An SSL certificate is a data file hosted in a website's origin server.
SSL certificates make SSL/TLS encryption possible, and they contain the website's public key and the website's identity, along with related information.

By providing the domain name of your deployment, Nebari will automatically generate a certificate for you based on the default `certificate` configuration below.
Nebari uses [Traefik](https://traefik.io/traefik/) to create and manage certificates.

The supported options are:

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<Tabs>
  <TabItem label="New self-signed" value="traefik" default="true">

By `default`, `Nebari` creates a [self-signed certificate](https://en.wikipedia.org/wiki/Self-signed_certificate).

```yaml
### Certificate configuration ###
certificate:
  type: self-signed
```

  </TabItem>
  <TabItem label="New Let's Encrypt" value="letsencrypt">

In order to create a certificate that's signed so that web browsers don't throw errors we currently support **Let's Encrypt**.

[Let’s Encrypt](https://letsencrypt.org/) is a CA.
In order to get a certificate for your website’s domain from Let’s Encrypt, Nebari requires extra information that abide by the [ACME protocol](https://tools.ietf.org/html/rfc8555) which typically runs on your web host.
This information is provided in the `letsencrypt` section of the configuration file.

```yaml
### Certificate configuration ###
certificate:
  type: lets-encrypt
  acme_email: <your-email-address>
  acme_server: https://acme-v02.api.letsencrypt.org/directory
```

You must specify:

- an email address that Let's Encrypt will associate the generated certificate with, and
- whether to use the [staging server](https://acme-staging-v02.api.letsencrypt.org/directory) or [production server](https://acme-v02.api.letsencrypt.org/directory).

In general you should use the production server, as seen above.

:::note
You can also generate the above configuration automatically by using the `--ssl-cert-email <your-email-address>` flag when you run `nebari init` to initialize your project.
:::

Let's Encrypt heavily rate limits their production endpoint. In order to avoid throttling, Nebari's traefik deployments will store retrieved certificates for the duration of their validity in a mounted PVC at a default location `/mnt/acme-certificates/acme.json`.

:::note
In order to refresh the certificate before it is invalidated, you will need to delete the `acme.json` file then restart the Traefik deployment by deleting the existing pod and letting a new one spin up. This may be necessary if you change the domain name of your Nebari deployment.
:::

  </TabItem>
  <TabItem label="Custom self-signed" value="Custom">

You may also supply a custom self-signed certificate and secret key.

```yaml
### Certificate configuration ###
certificate:
  type: existing
  secret_name: <secret-name>
```

To add the TLS certificate to Kubernetes run the following command with existing files.

```shell
kubectl create secret tls <secret-name> \
  --namespace=<namespace> \
  --cert=path/to/cert/file --key=path/to/key/file
```

:::note
The Kubernetes default namespace that Nebari uses is `dev`.
Otherwise, it will be your `namespace` defined in `nebari-config.yaml`.
:::

#### Wildcard certificates

Some of Nebari services might require special subdomains under your certificate, wildcard certificates allow you to secure all subdomains of a domain with a single certificate.
Defining a wildcard certificate decreases the amount of Common Name (CN) names you would need to define under the certificate configuration and reduces the chance of generating an incorrect subdomain.

</TabItem>
</Tabs>

### Shared Storage Configuration

:::note
As of Nebari 2024.9.1, alpha support for [Ceph](https://docs.ceph.com/en/latest/) shared file systems as an alternative to NFS is available.
:::

Nebari includes shared file systems for the jupyterhub user storage, jupyterhub shared storage, and conda store shared storage. By default, NFS drives are used.

The initial benefit of using Ceph is increased read/write performance compared to NFS, but further benefits are expected in future development. Ceph is a distributed storage system which has the potential to provide increased performance, high availability, data redundancy, storage consolidation, and scalability to Nebari.

:::danger
Do not switch from one storage type to another on an existing Nebari deployment. Any files in the user home directory and conda environments will be lost if you do so! On GCP, all node groups in the cluster will be destroyed and recreated. Only change the storage type prior to the initial deployment.
:::

Storage is configured in the `nebari-config.yaml` file under the storage section.

```yaml
storage:
  type: nfs
  conda_store: 200Gi
  shared_filesystem: 200Gi
```

Supported values for `storage.type` are `nfs` (default on most cloud providers), `efs` (default on AWS), and `cephfs`.

When using the `cephfs` storage type option, the block storage underlying all Ceph storage will be provisioned through the same Kubernetes storage class. By default, Kubernetes will use the default storage class unless a specific one is provided. For enhanced performance, some cloud providers offer premium storage class options.

You can specify the desired storage class under `ceph.storage_class_name` section in the configuration file. Below are examples of potential storage class values for various cloud providers:

<Tabs>
  <TabItem label="AWS" value="AWS" default="true">

Premium storage is not available on AWS.
</TabItem>
<TabItem label="Azure" value="Azure">

```yaml
ceph:
  storage_class_name: managed-premium
```

  </TabItem>
  <TabItem label="GCP" value="GCP">

```yaml
ceph:
  storage_class_name: premium-rwo
```

  </TabItem>
  <TabItem label="Existing" value="Existing">

```yaml
ceph:
  storage_class_name: some-cluster-storage-class
```

  </TabItem>
  <TabItem label="Local" value="Local">

Ceph is not supported on local deployments.
</TabItem>
</Tabs>

:::note
Premium storage is not available for some cloud providers on all node types. Check the documentation for your specific cloud provider to confirm which node types are compatible with which storage classes.
:::

## More configuration options

Learn to configure more aspects of your Nebari deployment with the following topic guides:

- [Security configuration](./advanced-security-configuration.md)
- [Cloud provider configuration](./advanced-provider-configuration.md)
- [JupyterLab and Dask profile configuration](./advanced-profiles-settings.md)
- [Customize JuputerHub theme](./advanced-custom-settings.md)
- [Environment configuration](./advanced-env-configuration.md)
- [Custom settings and overrides](./advanced-custom-settings.md)



---
File: /nebari-docs/docs/docs/explanations/advanced-custom-settings.md
---

---
title: Custom configurations and overrides
id: custom-overrides-configuration
description: Manage Terraform state, default images, storage, or override config.
---

## Custom settings

### Terraform state

Terraform manages the state of all the deployed resources via [backends](https://www.terraform.io/language/settings/backends). Terraform requires storing the state in order to keep
track of the names, ids, and states of deployed resources. See [terraform remote state](https://www.terraform.io/language/state/remote) for more information.

If you are doing anything other than testing we highly recommend `remote` unless you know what you
are doing. As any unexpected changes to the state can cause issues with the deployment.

For a `existing` provider that deploys to a kubernetes cluster, the kubernetes `remote` backend is also used.

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<Tabs>

<TabItem value="remote" label="Remote" default="true" >

```yaml
terraform_state:
  type: remote
```

</TabItem>

<TabItem value="local" label="Local" >

The simplest approach is storing the state on the local filesystem but this isn't recommended and isn't the default of Nebari.

```yaml
terraform_state:
  type: local
```

</TabItem>

<TabItem value="existing" label="Existing" >

Using an existing terraform backend can be done by specifying the `backend` and arbitrary key/value pairs in the `config`.

```yaml
terraform_state:
  type: existing
  backend: s3
  config:
    bucket: mybucket
    key: "path/to/my/key"
    region: "us-east-1"
```

</TabItem>

</Tabs>

### Default Images

Nebari uses Docker images to provide containers for JupyterHub, JupyterLab interface, and the Dask worker user environments.

Default images are the image run by default if not explicitly specified in a profile (described in the next section).

The `jupyterhub` key controls the JupyterHub image run.

These control the docker image used to run JupyterHub, the default JupyterLab image, and the default Dask worker image respectively.

```yaml
### Default images ###
default_images:
  jupyterhub: "quay.io/nebari/nebari-jupyterhub:v2022.10.1"
  jupyterlab: "quay.io/nebari/nebari-jupyterlab:v2022.10.1"
  dask_worker: "quay.io/nebari/nebari-dask-worker:v2022.10.1"
```

### Storage

The Storage section is used to control the amount of storage allocated to the **shared filesystem**.

```yaml
### Storage ###
storage:
  conda_store: 200Gi
  shared_filesystem: 200Gi
```

:::warning
For most providers, when the storage size is changed, it will automatically **delete** the previous storage place.

Changing the storage size on an AWS deployment after the initial deployment can be especially tricky so it might be worthwhile padding these storage sizes.
:::

## Overrides

Overrides allows you to override the default configuration for a given resource on Nebari without having to directly modify the infrastructure components.

Below we show the available resources that can be overridden in the configuration.

### JupyterHub

JupyterHub uses the [zero to jupyterhub helm chart](https://github.com/jupyterhub/zero-to-jupyterhub-k8s/). This chart has many options that are not configured in the Nebari default
installation. You can override specific values in the [values.yaml](https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/jupyterhub/values.yaml). `jupyterhub.overrides`
is optional.

```yaml
jupyterhub:
  overrides:
    cull:
      users: true
```

### JupyterLab

Nebari supports a number of configuration options for JupyterLab:

- `jupyterlab.idle_culler` - This is used to configure the idle culler for JupyterLab. See [idle culling](/docs/how-tos/idle-culling) for more information.

```yaml
jupyterlab:
  idle_culler:
    kernel_cull_idle_timeout: 30
```

- `jupyterlab.initial_repositories` - Auto-deploys specified git repositories by cloning them into user directories upon JupyterLab instance initialization. Accepts a list of `name: url` pairs, with each `name` becoming the folder name in the user's home directory. Once the repository is cloned in the user space, it will not be updated. In order to update to the latest repository, users must delete the folder and restart the server.

```yaml
jupyterlab:
  initial_repositories:
    - examples/nebari-demo: https://github.com/nebari-dev/nebari-demo.git
```

:::note Note
Path location key should not start or end with trailing slash.
You can configure JupyterLab to open in a location within the cloned repository by setting `preferred_dir` option within the `jupyterlab` group.
:::

See also `jupyterlab.gallery_settings` (documented below) which defers the cloning of repositories until user requests it and provides a rich presentation layer.

:::warning
While you could embed an access token in the URL to fetch from a private repository, please beware that this token can be accessed by each user - you should only use tightly scoped personal access tokens which you are comfortable to share with each of your users.
:::

- `jupyterlab.default_settings` - Enables overriding the default JupyterLab and JupyterLab extensions settings. Users will still be able to adjust the settings in the JupyterLab Setting Editor. The keys should be names of the Jupyter plugins with values defining mapping between the plugin setting and new default.

```yaml
jupyterlab:
  default_settings:
    "@jupyterlab/apputils-extension:themes":
      theme: JupyterLab Dark
```

- `jupyterlab.preferred_dir` - Sets the default location in which JupyterLab should open the file browser in.

- `jupyterlab.gallery_settings` - Configures [`jupyterlab-gallery`](https://github.com/nebari-dev/jupyterlab-gallery) extension which enables user to clone (and later synchronise) pre-specified repositories.

```yaml
jupyterlab:
  gallery_settings:
    title: Example repositories
    destination: examples
    exhibits:
      - title: Nebari
        git: https://github.com/nebari-dev/nebari.git
        homepage: https://github.com/nebari-dev/nebari
        description: 🪴 Nebari - your open source data science platform
      - title: PyTorch Tutorial
        git: https://github.com/your-org/your-repository.git
        account: name_of_dedicated_account
        token: YOUR_PERSONAL_ACCESS_TOKEN
        icon: https://your.domain/icon.svg
```

:::warning
While private repositories can be cloned by passing `account` and `token`, the access token can be accessed by each user - you should only use tightly scoped personal access tokens which you are comfortable to share with each of your users.
:::

### Terraform

The Nebari configuration file provides a huge number of configuration options for customizing your Nebari infrastructure. While these options are sufficient for an average user, they
aren't exhaustive by any means. There are still a plenty of things you might want to achieve which cannot be configured directly by the above mentioned options. Therefore, we've
introduced a new option called terraform overrides (`terraform_overrides`), which lets you override the values of terraform variables in specific modules/resource. This is a
relatively advanced feature and must be used with utmost care, and you should really know what you're doing.

Here we describe the overrides supported via Nebari config file:

#### Ingress

You can configure the IP of the load balancer and add annotations for the same via `ingress`'s terraform overrides, one such example for GCP is:

```yaml
ingress:
  terraform_overrides:
      load-balancer-annotations:
          "networking.gke.io/load-balancer-type": "Internal"
          "networking.gke.io/internal-load-balancer-subnet": "pre-existing-subnet"
      load-balancer-ip: "1.2.3.4"
```

This is quite useful for pinning the IP Address of the load balancer.

#### Deployment inside a Virtual Private Network

<Tabs>

<TabItem value="azure" label="Azure" default="true" >

You can deploy your cluster into a Virtual Private Network (VPN) or Virtual Network (VNET).

An example configuration for Azure is given below:

```yaml
azure:
  region: Central US
  ...
  vnet_subnet_id: '/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.Network/virtualNetworks/<vnet-name>/subnets/<subnet-name>'
```

If you want the ASK cluster to be [private cluster](https://learn.microsoft.com/en-us/azure/ask/private-clusters?tabs=azure-portal).

For extra security, you can deploy your cluster from an [Azure Bastion host](https://learn.microsoft.com/en-us/azure/ask/operator-best-practices-network#securely-connect-to-nodes-through-a-bastion-host) (or jump host), making the Kubernetes API only accessible from this one secure machine. You will likely need to also modify the network_profile as follows:

```yaml
azure:
  region: Central US
  private_cluster_enabled: true
  vnet_subnet_id: '/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.Network/virtualNetworks/<vnet-name>/subnets/<subnet-name>'
  network_profile:
    service_cidr: "10.0.2.0/24" # how many IPs would you like to reserve for Nebari
    network_plugin: "azure"
    network_policy: "azure"
    dns_service_ip: "10.0.2.10" # must be within the `service_cidr` range from above
    docker_bridge_cidr: "172.17.0.1/16" # no real need to change this

```

</TabItem>

<TabItem value="gcp" label="GCP" default="true" >

You can also deploy inside a [Virtual Private Cloud (VPC) in GCP](https://cloud.google.com/vpc/docs/overview), making the Kubernetes cluster private. Here is an example configuration:

```yaml
google_cloud_platform:
  networking_mode: "VPC_NATIVE"
  network: "your-vpc-name"
  subnetwork: "your-vpc-subnet-name"
  private_cluster_config:
    enable_private_nodes: true
    enable_private_endpoint: true
    master_ipv4_cidr_block: "172.16.0.32/28"
  master_authorized_networks_config:
    cidr_block: null
    display_name: null
```

As the name suggests the cluster will be private, which means it would not have access to the internet - not ideal for deploying pods in the cluster. Therefore, we need
to allow internet access for the cluster, which can be achieved by creating a [Network Address Translation](https://en.wikipedia.org/wiki/Network_address_translation) router. The following two commands are an example of how you can do this for your VPC network on GCP.

```bash
gcloud compute routers create nebari-nat-router --network your-vpc-name --region your-region

gcloud compute routers nats create nat-config --router nebari-nat-router  --nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips --region your-region
```

</TabItem>

</Tabs>

Deployment inside a virtual network is slightly different from deploying inside a public network.
As the name suggests, since it's a virtual private network, you need to be inside the network to able to deploy and access Nebari.
One way to achieve this is by creating a Virtual Machine (VM) inside the virtual network.
Select the virtual network and subnet name under the networking settings of your cloud provider while creating the VM
and then follow the usual deployment instructions as you would deploy from your local machine.

=======

#### Conda store worker

You can use the following settings to change the defaults settings (shown) used for Conda store workers.

```yaml
conda_store:
  max_workers: 50
  worker_resources:
    requests:
      cpu: 1
      memory: 4Gi
```

:::note Note
Current `conda_store.worker_resources` defaults are set at the minimum recommended resources for conda-store-workers - (conda-store [docs](https://conda.store/conda-store/references/faq#what-are-the-resource-requirements-for-conda-store-server))
:::

## Helm Extensions

Nebari provides a way for any user to expand the infrastructure available by default by using the `helm_extensions` attribute. This attribute allows for the management and customization of Kubernetes applications through Helm charts. The helm_extensions is a configuration construct that specifies a list of Helm charts and their respective settings.

### Overview

Each entry in the `helm_extensions` list represents a single Helm chart configuration, allowing you to define the chart source, version, and specific overrides or settings for that chart. When Nebari is deployed, it will install the specified Helm charts using the provided settings.

### Structure

Each entry in the helm_extensions list contains the following fields:

- `name`: A unique identifier for the Helm chart. It will also be used as the name of the Kubernetes deployment related resources.
- `repository`: The URL of the repository where the Helm chart is stored. Must be a valid Helm repository URL.
- `chart`: The name or path of the chart within the repository. must be compliant with the Helm chart naming conventions.
- `version`: The specific version of the chart to be used.
- `overrides`: Specific configurations to override default chart values.

:::note Note
The `overrides` field is optional. If not specified, the default values for the chart will be used.
:::

### Example

Below we give an example showcasing how to install Redis using helm_extensions:

```yaml
helm_extensions:
  - name: redis-deployment
    repository: https://charts.bitnami.com/bitnami
    chart: redis
    version: 17.7.5
    overrides:
      architecture: standalone
      master:
        containerSecurityContext:
          runAsUser: 0
        persistence:
          enabled: true
          path: /redis/data
          subPath: redis/data
          existingClaim: <existing-claim-name-is-required>
      replica:
        persistence:
          enabled: false
        replicaCount: 0
```

:::warning Warning
In the above example, we are assuming the current nebari kubernetes cluster already has an appropriate storage class and persistent volume claim (PVC) created. If not, you will need to create a storage class and PVC before deploying the helm chart.
:::



---
File: /nebari-docs/docs/docs/explanations/advanced-customize-themes.md
---

---
title: Customize JupyterHub theme
id: customize-themes
description: Customize colors, welcome message, and more.
---

# Customize JupyterHub theme

Nebari uses a custom JupyterHub theme, [nebari-dev/nebari-jupyterhub-theme](https://github.com/nebari-dev/nebari-jupyterhub-theme). Users can further customize the theme through these available options:

Below is an example of a custom theme configuration for a GCP deployment:

```yaml
### Theme ###
theme:
  jupyterhub:
    hub_title: demo.nebari.dev
    hub_subtitle: Your open source data science platform, hosted on Google Cloud Platform
    welcome: |
      Welcome! Learn about Nebari's features and configurations in <a href="https://www.nebari.dev/docs">the
      documentation</a>. If you have any questions or feedback, reach the team on
      <a href="https://www.nebari.dev/docs/community#getting-support">Nebari's support
      forums</a>.
    logo: /hub/custom/images/Nebari-Logo-Horizontal-Lockup-White-text.svg
    primary_color: "#cb39ed"
    secondary_color: "#2bd1c5"
    accent_color: "#eda61d"
    text_color: "#1c1d26",
    h1_color: "#0f1015",
    h2_color: "#0f1015",
    navbar_text_color: "#E8E8E8",
    narbar_hover_color: "#00a3b0",
    navbar_color: "#1c1d26",
    display_version: true
    version: v2022.10.1
```

![A demonstration os the theming customizations](/img/how-tos/nebari_login_screen.png)

:::note
To properly update the image logo, you must ensure that the provided logo field contains an accessible path to the logo. This file must be added in the JupyterHub image, or by passing a valid URL to the logo.
:::



---
File: /nebari-docs/docs/docs/explanations/advanced-env-configuration.md
---

---
title: Environment configurations
id: environments-configuration
description: Configure environments with conda-store
---

# Environment configurations

:::tip
For guidance on managing or creating environments directly with conda-store, see our guide on [Creating new environments on Nebari](/docs/tutorials/creating-new-environments.md).
If you're looking for information on updating your namespaces after a recent upgrade, refer to our troubleshooting guide on [Migrating namespaces](/docs/troubleshooting.mdx#conda-store-compatibility-migration-steps-when-upgrading-to-045).
:::

<!-- TODO: Update the time to create environments based in new conda-store updates -->

Nebari comes with two default environments that are built during deployment:

- `nebari-git-nebari-git-dask` to [run distributed workflows with Dask][using-dask],
- `nebari-git-nebari-git-dashboard` to [create shareable dashboard][create-dashboards].

The configuration of each environment in Nebari is achieved through a `environment.<filename>` mapping to a conda environment specification. To configure environments, you can add entries to the `nebari-config.yml` file located in the deployment repository, which will then be used by conda-store to create or update the environment during deployment.

For example, the following snippet shows an environment configuration in `nebari-config.yml`:

```yaml
### Example environment configuration
environments:
  "environment-example.yaml":
    name: example
    channels:
      - conda-forge
      - defaults
    dependencies:
      - python=3.9
      - ipykernel
      - ipywidgets
      - numpy
      - numba
      - pandas
      - jinja2
      - pyyaml
```

In this example, the environment is defined with the name "example" and includes various dependencies specified as a list of package names and versions. Additionally, two channels, `conda-forge` and `defaults`, are specified to be used to retrieve the packages during environment creation.

:::note
One current requirement is that each environment _must_ include `ipykernel` and `ipywidgets` to properly show up in the JupyterLab environment.
:::

After you modify an environment definition in the configuration, it may take 1-10 minutes for the changes to take effect after you deploy the configuration. Once the environment is updated, you can access it from the JupyterLab environment selection menu under the `nebari-git` namespace. For example, if you created an environment called `example` (as show in the above example), it will be available as `nebari-git-nebari-git-example`.

:::note
The double `nebari-git` is a known consequence of using conda-store for environment management.

Learn more in the [FAQ: Why do I see duplication in the names of environments?](../faq#why-is-there-duplication-in-names-of-environments)
:::

:::warning
For versions of Nebari earlier than `2023.04.1`, conda-store by default restricts the environment channels to only accept `defaults` and `conda-forge`.

<!-- If you want to use other channels, you can follow the instructions in [Managing conda environment][]. -->

:::

<!-- Internal links -->

[using-dask]: ../tutorials/using_dask

### Built-in namespaces:

In Nebari, namespaces are used to organize and isolate resources. Nebari has two built-in namespaces: `nebari-git` and `global`.

The `nebari-git` namespace refers to all available environments created using the `nebari-config.yaml` file and is available for all users and services. On the other hand, the global namespace (previously known as default) is the default namespace used by conda-store to manage its internal components and workers. It is designed for environments that are specific to a user or a service.

For more information, please refer to conda-store [administration documentation](https://conda.store/en/latest/administration.html) related to `CondaStore.default_namespace` and `CondaStore.filesystem_namespace` respectively.

:::note
By default, the namespace `nebari-git` is used in a standard Nebari deployment. However, you can change the namespace name by modifying the `conda-store.default_namespace` entry in your `nebari-config.yaml` configuration file. Keep in mind that this setting permanently changes the namespace name.
:::

When you specify an environment in `nebari-config.yml`, it will be made available for all users and services under the `nebari-git` namespace. Conda-store is responsible for creating them upon request from the deployment process.

However, the conda-store permission model restricts user intervention for both namespaces. This means that `nebari-git` environments can only be modified within the Nebari deployments, while global environments can be locally changed via direct interaction with conda activate. However, any changes made to `global` environments will only be perceived by the user and not propagated to other users.

:::warning
To reiterate, the `nebari-git` namespace is modified using nebari-config deployments only, and any modifications to the `global` namespace will only be available locally.
:::

:::note
While it is not possible to interact with these namespaces directly, it is possible to override the conda-store permission model to allow users to modify the environments.
For more information please refer to [Handle Access to restricted namespaces](/docs/troubleshooting.mdx#handle-access-to-restricted-namespaces)
:::



---
File: /nebari-docs/docs/docs/explanations/advanced-profiles-settings.md
---

---
title: Configure profiles
id: profile-configuration
description: Customize JupyterLab and Dask environments.
---

# Configure profiles

Profiles are used to control the JupyterLab user instances and Dask workers provided by Dask Gateway.

## JupyterLab profiles

```yaml
### JupyterLab Profiles ###
profiles:
  jupyterlab:
    - display_name: Small Instance
      description: Stable environment with 2 cpu / 8 GB ram
      access: all
      default: true
      profile_options:  # optional
        image:
          display_name: Image
          choices:
            default:
              display_name: nebari-jupyterlab:latest
              default: true
              kubespawner_override:
                image: quay.io/nebari/nebari-jupyterlab:latest
            custom:
              display_name: my-custom-image:mytag
              kubespawner_override:
                image: <my-container-registry>/myOrg/my-custom-image:mytag
      kubespawner_override:
        cpu_limit: 2
        cpu_guarantee: 1.5
        mem_limit: 8G
        mem_guarantee: 6G
    - display_name: Medium Instance
    ...
```

![Default profiles showing small, medium, and large instances](/img/explanations/profiles-server-options.png)

Each profile under `jupyterlab` is a named JupyterLab profile.

`display_name` is the name of the profile that will be displayed to users.

`description` is a description of the profile that will be displayed to users.

`profile_options` makes it possible to set various sub-options per profile. See the [Kubespawner docs](https://jupyterhub-kubespawner.readthedocs.io/en/latest/spawner.html#kubespawner.KubeSpawner.profile_list) for more info.

`kubespawner_override` field to define behavior as per the [KubeSpawner](https://jupyterhub-kubespawner.readthedocs.io/en/latest/spawner.html) API.

It is possible to control which users have access to which JupyterLab profiles. Each profile has a field named `access` which can be set to `all` (default if omitted), `yaml`, or
`keycloak`.

- `all` means every user will have access to the profile (default).
- `yaml` means that access is restricted to anyone with their username in the `users` field of the profile or who belongs to a group named in the `groups` field.
- `keycloak` means that access is restricted to any user who in Keycloak has either their group(s) or user with the attribute `jupyterlab_profiles` containing this profile name. For
  example, if the user is in a Keycloak group named `developers` which has an attribute `jupyterlab_profiles` set to `Large Instance`, they will have access to the Large Instance
  profile. To specify multiple profiles for one group (or user) delimit their names using `##` - for example, `Large Instance##Another Instance`.

When configuring the memory and CPUs for profiles, there are some important considerations to make. Two important terms to understand are:

- `limit`: the absolute max memory that a given pod can consume. If a process within the pod consumes more than the `limit` memory the Linux OS will kill the process. Limit is not
  used for scheduling purposes with kubernetes.
- `guarantee`: is the amount of memory the kubernetes scheduler uses to place a given pod. In general the `guarantee` will be less than the limit. Often times the node itself has
  less available memory than the node specification. See this [guide from digital ocean](https://docs.digitalocean.com/products/kubernetes/details/limits/#allocatable-memory) which is generally
  applicable to other clouds.

For example if a node has 8 GB of ram and 2 CPUs you should guarantee/schedule roughly 75% and follow the digital ocean guide linked above, e.g. 1.5 CPU guaranteed and 5.5 GB guaranteed.

### JupyterLab Profile Node Selectors

A common operation is to target jupyterlab profiles to specific node labels. In order to target a specific node groups add the following.
This example shows a GKE node groups with name `user-large`.
Other cloud providers will have different node labels.

```yaml
### JupyterLab Profiles ###
profiles:
  jupyterlab:
    - display_name: Small Instance
      ...
      kubespawner_override:
        ...
        node_selector:
          "cloud.google.com/gke-nodepool": "user-large"
        ...
```

### Specifying GPU/Accelerator Requirements

If you want to ensure that you have GPU resources use the following annotations.

```yaml
### JupyterLab Profiles ###
profiles:
  jupyterlab:
    - display_name: Small Instance
      ...
      kubespawner_override:
        ...
        extra_resource_limits:
          nvidia.com/gpu: 1
        ...
```

## Dask profiles

Finally, we allow for configuration of the Dask workers. In general, similar to the JupyterLab instances you only need to configure the cores and memory.

```yaml
### Dask Profiles ###
profiles:
  ...
  dask_worker:
    "Small Worker":
      worker_cores_limit: 1
      worker_cores: 1
      worker_memory_limit: 1G
      worker_memory: 1G
    "Medium Worker":
      worker_cores_limit: 1.5
      worker_cores: 1.25
      worker_memory_limit: 2G
      worker_memory: 2G
```

### Dask Scheduler

In a few instances, the Dask worker node-group might be running on quite a large instance, perhaps with 8 CPUs and 32 GB of memory (or more). When this is the case, you might also
want to increase the resource levels associated with the Dask Scheduler.

```yaml
### Dask Profiles ###
profiles:
  ...
  dask_worker:
      "Huge Worker":
        worker_cores_limit: 7
        worker_cores: 6
        worker_memory_limit: 30G
        worker_memory: 28G
        scheduler_cores_limit: 7
        scheduler_cores: 6
        scheduler_memory_limit: 30G
        scheduler_memory: 28G
```



---
File: /nebari-docs/docs/docs/explanations/advanced-provider-configuration.md
---

---
title: Cloud provider configuration
id: provider-configuration
description: Configure region, instance types, and more.
---

# Provider configuration

To take advantage of the auto-scaling and dask-distributed computing capabilities, Nebari can be deployed on a handful of the most commonly used cloud providers.
Nebari utilizes many of the resources these cloud providers have to offer;
however, the Kubernetes engine (or service) is at it's core.
Each cloud provider has slightly different ways that Kubernetes is configured but fear not, all of this is handled by Nebari.

The `provider` section of the configuration file allows you to configure the cloud provider that you are deploying to.
Including the region, instance types, and other cloud specific configurations.

Select the provider of your choice:

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<Tabs>

<TabItem value="gcp" label="GCP" default="true" >

[Google Cloud](https://cloud.google.com/) has the best support for Nebari and is a great default choice for a production deployment. It allows auto-scaling to zero within the node group. There are no major
restrictions.

To see available instance types refer to [GCP docs](https://cloud.google.com/compute/docs/machine-types).

:::note
By default the [GKE release channel](https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels) is set to `UNSPECIFIED` to prevent the cluster from auto-updating. This has the advantage of ensuring that the Kubernetes version doesn't upgrade and potentially introduce breaking changes. If you'd prefer your cluster's Kubernetes version to update automatically, you can specify a release channel; the options are either `stable`, `regular` or `rapid`.
:::

```yaml
### Provider configuration ###
google_cloud_platform:
  project: test-test-test
  region: us-central1
  kubernetes_version: "1.24.11-gke.1000"
  release_channel: "UNSPECIFIED"  # default is hidden
  node_groups:
    general:
      instance: n1-standard-4
      min_nodes: 1
      max_nodes: 1
    user:
      instance: n1-standard-2
      min_nodes: 0
      max_nodes: 5
    worker:
      instance: n1-standard-2
      min_nodes: 0
      max_nodes: 5
```

</TabItem>

<TabItem value="aws" label="AWS">

[Amazon Web Services](https://aws.amazon.com/) also has support for auto-scaling to zero within the node group with no restrictions.

Consult [AWS instance types](https://aws.amazon.com/ec2/instance-types/) for possible options.

```yaml
### Provider configuration ###
amazon_web_services:
  region: us-west-2
  kubernetes_version: "1.18"
  node_groups:
    general:
      instance: "m5.xlarge"
      min_nodes: 1
      max_nodes: 1
    user:
      instance: "m5.large"
      min_nodes: 1
      max_nodes: 5
    worker:
      instance: "m5.large"
      min_nodes: 1
      max_nodes: 5
```

### Permissions boundary (Optional)

Permissions boundaries in AWS is a powerful feature designed to control the maximum permissions a
user or role can have within an AWS Identity and Access Management (IAM) policy. By setting a permissions
boundary, administrators can enforce restrictions on the extent of permissions that can be granted,
even if policies would otherwise allow broader access.

Nebari supports setting permissions boundary while deploying Nebari to be applied on all the policies
created by Nebari. Here is an example of how you would set permissions boundary in `nebari-config.yaml`.

```yaml
amazon_web_services:
  # the arn for the permission's boundary policy
  permissions_boundary: arn:aws:iam::01234567890:policy/<permissions-boundary-policy-name>
```

### EKS KMS ARN (Optional)

You can use AWS Key Management Service (KMS) to enhance security by encrypting Kubernetes secrets in
Amazon Elastic Kubernetes Service (EKS). This approach adds an extra layer of protection for sensitive
information, like passwords, credentials, and TLS keys, by applying user-managed encryption keys to Kubernetes
secrets, supporting a [defense-in-depth strategy](https://aws.amazon.com/blogs/containers/using-eks-encryption-provider-support-for-defense-in-depth/).

Nebari supports setting an existing KMS key while deploying Nebari to implement encryption of secrets
created in Nebari's EKS cluster. The KMS key must be a **Symmetric** key set to **encrypt and decrypt** data.

:::warning
Enabling EKS cluster secrets encryption, by setting `amazon_web_services.eks_kms_arn`, is an
_irreversible_ action and re-deploying Nebari to try to remove a previously set `eks_kms_arn` will fail.
On the other hand, if you try to change the KMS key in use for cluster encryption, by re-deploying Nebari
after setting a _different_ key ARN, the re-deploy should succeed but the KMS key used for encryption will
not actually change in the cluster config and the original key will remain set. The integrity of a faulty
deployment can be restored, following a failed re-deploy attempt to remove a previously set KMS key, by
simply re-deploying Nebari while ensuring `eks_kms_arn` is set to the original KMS key ARN.
:::

:::danger
If the KMS key used for envelope encryption of secrets is ever deleted, then there is no way to recover
the EKS cluster.
:::

:::note
After enabling cluster encryption on your cluster, you must encrypt all existing secrets with the
new key by running the following command:
`kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - kms-encryption-timestamp="time value"`
Consult [Encrypt K8s secrets with AWS KMS on existing clusters](https://docs.aws.amazon.com/eks/latest/userguide/enable-kms.html) for more information.
:::

Here is an example of how you would set KMS key ARN in `nebari-config.yaml`.

```yaml
amazon_web_services:
  # the arn for the AWS Key Management Service key
  eks_kms_arn: "arn:aws:kms:us-west-2:01234567890:key/<aws-kms-key-id>"
```

### Launch Templates (Optional)

Nebari supports configuring launch templates for your node groups, enabling you to customize settings like the AMI ID and pre-bootstrap commands. This is particularly useful if you need to use a custom AMI or perform specific actions before the node joins the cluster.

:::warning
If you add a `launch_template` to an existing node group that was previously created without one, AWS will treat this as a change requiring the replacement of the entire node group. This action will trigger a reallocation of resources, effectively destroying the current node group and recreating it. This behavior is due to how AWS handles self-managed node groups versus those using launch templates with custom settings.
:::

:::tip
To avoid unexpected downtime or data loss, consider creating a new node group with the launch template settings and migrating your workloads accordingly. This approach allows you to implement the new configuration without disrupting your existing resources.
:::

#### Configuring a Launch Template

To configure a launch template for a node group in your `nebari-config.yaml`, add the `launch_template` section under the desired node group:

```yaml
amazon_web_services:
  region: us-west-2
  kubernetes_version: "1.18"
  node_groups:
    custom-node-group:
      instance: "m5.large"
      min_nodes: 1
      max_nodes: 5
      gpu: false  # Set to true if using GPU instances
      launch_template:
        # Replace with your custom AMI ID
        ami_id: ami-0abcdef1234567890
        # Command to run before the node joins the cluster
        pre_bootstrap_command: |
          #!/bin/bash
          # This script is executed before the node is bootstrapped
          # You can use this script to install additional packages or configure the node
          # For example, to install the `htop` package, you can run:
          # sudo apt-get update
          # sudo apt-get install -y htop"
```

**Parameters:**

- `ami_id` (Optional): The ID of the custom AMI to use for the nodes in this group; this assumes the AMI provided is an EKS-optimized AMI derivative. If specified, the `ami_type` is automatically set to `CUSTOM`.
- `pre_bootstrap_command` (Optional): A command or script to execute on the node before
  it joins the Kubernetes cluster. This can be used for custom setup or configuration
  tasks. The format should be a single string in conformation with the shell syntax.
  This command is injected in the `user_data` field of the launch template. For more
  information, see [User Data](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html).

> If you're using a `launch_template` with a custom `ami_id`, there's an issue with updating the `scaling.desired_size` via Nebari configuration (terraform). To scale up, you must recreate the node group or adjust the scaling settings directly in the AWS Console UI (recommended). We are aware of this inconsistency and plan to address it in a future update.

:::note
If an `ami_id` is not provided, AWS will use the default Amazon Linux 2 AMI for the
specified instance type. You can find the latest optimized AMI IDs for Amazon EKS in your
cluster region by inspecting its respective SSM parameters. For more information, see
[Retrieve recommended Amazon Linux AMI IDs](https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html).
:::

</TabItem>

<TabItem value="azure" label="Azure">

[Microsoft Azure](https://azure.microsoft.com/) has similar settings for Kubernetes version, region, and instance names - using Azure's available values of course.

Azure also requires a field named `storage_account_postfix` which will have been generated by `nebari init`. This allows nebari to create a Storage Account bucket that should be globally unique.

```yaml
### Provider configuration ###
azure:
  region: Central US
  kubernetes_version: 1.19.11
  node_groups:
    general:
      instance: Standard_D4_v3
      min_nodes: 1
      max_nodes: 1
    user:
      instance: Standard_D2_v2
      min_nodes: 0
      max_nodes: 5
    worker:
      instance: Standard_D2_v2
      min_nodes: 0
      max_nodes: 5
  storage_account_postfix: t65ft6q5
```

</TabItem>

<TabItem value="existing" label="Existing Kubernetes clusters">

Originally designed for Nebari deployments on a "local" minikube cluster, this feature has now expanded to allow users to deploy Nebari to any existing kubernetes cluster.
The default options for an `existing` deployment are still set to deploy to a minikube cluster.

<!-- TODO: Uncomment and add link when available -->
<!-- If you wish to deploy to an existing kubernetes cluster on one of the cloud providers, please refer to a more detailed walkthrough found in the [Deploy Nebari to an Existing Kubernetes Cluster]. -->

Deploying to a local existing kubernetes cluster has different options than the cloud providers. `kube_context` is an optional key that can be used to deploy to a non-default context.
The default node selectors will allow pods to be scheduled anywhere. This can be adjusted to schedule pods on different labeled nodes, allowing for similar functionality to node groups in the cloud.

```yaml
### Provider configuration ###
existing:
  kube_context: minikube
  node_selectors:
    general:
      key: kubernetes.io/os
      value: linux
    user:
      key: kubernetes.io/os
      value: linux
    worker:
      key: kubernetes.io/os
      value: linux
```

</TabItem>

<TabItem value="local" label="Local (testing)">

Local deployment is intended for Nebari deployments on a "local" cluster created and management by Kind.
It is great for experimentation and development.

:::warning
Currently, local mode is only supported for Linux-based operating systems.
:::

```yaml
### Provider configuration ###
local:
  kube_context: minikube
  node_selectors:
    general:
      key: kubernetes.io/os
      value: linux
    user:
      key: kubernetes.io/os
      value: linux
    worker:
      key: kubernetes.io/os
      value: linux
```

</TabItem>

</Tabs>

:::note
Many of the cloud providers regularly update their internal **Kubernetes versions** so if you wish to specify a particular version, please check the following resources.
This is _completely optional_ as Nebari will, by default, select the most recent version available for your preferred cloud provider:
[Google Cloud Platform](https://cloud.google.com/kubernetes-engine/docs/release-notes-stable);
[Amazon Web Services](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html);
[Microsoft Azure](https://docs.microsoft.com/en-us/azure/ask/supported-kubernetes-versions?tabs=azure-cli).
:::



---
File: /nebari-docs/docs/docs/explanations/advanced-security-configuration.md
---

---
title: Security configuration
id: security-configuration
description: Guide to configure Keycloak and authentication methods.
---

# Security configuration

The `security` section of the configuration file allows you to configure the authentication and authorization providers of your deployment.
As well as customize the default configurations of the Keycloak User management system.

## Keycloak

[Keycloak](https://www.keycloak.org/) is an open source identity and access management solution that provides authentication, authorization, and user management for web, mobile, IoT, and internal applications.
This section outlines the configuration options for the Keycloak service that Nebari provides.

```yaml
### Keycloak configuration ###
security:
  keycloak:
    initial_root_password: initpasswd
    overrides:
      image:
        repository: quansight/nebari-keycloak
  ...
```

The `keycloak` section allows you to specify an initial password for the `root` Administrative user to manage your Keycloak database which is responsible for managing users, clients, and other Keycloak related configurations. Note that the `root` user is not actually a Nebari user - you cannot access the
main features of Nebari such as JupyterLab with this user - it is exclusively for Keycloak management.

The `overrides` section allows you to specify a custom image for the Keycloak service.
This is useful if you want to customize themes or add additional plugins to Keycloak.
The full extent of override options can be found in the [Keycloak Helm deployment](https://github.com/codecentric/helm-charts/tree/master/charts/keycloak).

:::warning
We strongly recommend changing the `initial_root_password` after your initial deployment and deleting this value from your `nebari-config.yaml`.
Any changes to this value in the `nebari-config.yaml` after the initial deployment will have no effect.

For more information on how to do this, see the [Change Keycloak root password section](../how-tos/configuring-keycloak#change-keycloak-root-password).
:::

## Authentication methods

Nebari supports multiple authentication methods by using [Keycloak](https://www.keycloak.org/) under the hood.
To ease the configuration procedure of adding the most common authentication providers to Keycloak, Nebari already supports `[Auth0, GitHub, password]` automatically during deployment.
You may also disable authentication by setting `authentication` to `false` in the `nebari-config.yaml` file.

The default authentication method is set to `GitHub` if no changes are specified in the configuration file or during initialization.

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<Tabs>

<TabItem label="GitHub" value="github" default>

To use GitHub as your authentication method, you must first create a [GitHub OAuth application](https://docs.github.com/en/developers/apps/building-oauth-apps/creating-an-oauth-app) and provide the `client_id` and `client_secret` to Nebari.
By using the `GitHub` authentication method, users will then be able to log in to Nebari using their GitHub account registered within Keycloak database.

```yaml
### Authentication configuration ###
security:
  ...
  authentication:
    type: GitHub
    config:
      client_id: ...
      client_secret: ...
```

</TabItem>

<TabItem label="Auth0" value="auth0">

Auth0 is a cloud-based identity management platform that provides authentication, authorization, and user management for web, mobile, IoT, and internal applications. This authentication method is useful for organizations that already have an Auth0 account and user database and want to seamlessly integrate it with Nebari.

To use Auth0 as your authentication method, you must have an [Auth0 application](https://auth0.com/docs/applications/set-up-an-application/register-single-page-app) and provide the `client_id` and `client_secret` to Nebari.
Make sure that your Auth0 application is a `Regular Web Application`.
By using the `Auth0` authentication method, users will then be able to log in to Nebari using their Auth0 account registered within Keycloak database.

```yaml
### Authentication configuration ###
security:
  ...
  authentication:
    type: Auth0
    config:
      client_id: ...
      client_secret: ...
      auth0_subdomain: ...
```

It's important to note is that the `auth0_subdomain` field in the YAML must be only the `<auth0_subdomain>.auth0.com`.
For example, for `nebari-dev.auth0.com` the subdomain would be `nebari-dev`.

:::note

Nebari supports automatic provisioning of the Auth0 application during initialization. To do so, you must provide the `--auth-provider=auth0 --auth-auto-provision` flags when running `nebari init`. This will automatically provide the `client_id` and `client_secret` to Nebari given that your Auth0 environment variables are set:

- `AUTH0_CLIENT_ID`: client ID of Auth0 machine-to-machine application found at top of the newly created application page
- `AUTH0_CLIENT_SECRET`: secret ID of Auth0 machine-to-machine application found in the `Settings` tab of the newly created application
- `AUTH0_DOMAIN`: The `Tenant Name` which can be found in the general account settings on the left hand side of the page appended with `.auth0.com`

:::

</TabItem>

<TabItem label="password" value="password" default="true">

Username and Password is the simplest authentication method that Nebari supports. By using the `Password` authentication method, users will then be able to log in to Nebari using their username and password registered within Keycloak database.

```yaml
### Authentication configuration ###
security:
  ...
  authentication:
    type: password
```

</TabItem>

</Tabs>

:::warning
The options for `type`, which are `Auth0`, `GitHub`, and `password`, are case sensitive.
:::

:::note
Even if you formally select `password/GitHub/Auth0` authentication in the `nebari-config.yaml` file, it's still possible to add other authentication methods alongside them to Keycloak manually.
For more information on how to do this, please refer to the [Keycloak documentation](https://www.keycloak.org/docs/latest/server_admin/index.html#_identity_broker).
:::



---
File: /nebari-docs/docs/docs/explanations/architecture.mdx
---

---
id: infrastructure-architecture
title: Nebari architecture
---

# Nebari architecture

Nebari relies on a few core concepts to make it work.
This guide will help you understand how Nebari works and how to use it.

Some of the most important concepts that Nebari relies on are Terraform and Kubernetes.

![Nebari core concepts](/img/explanations/core-concepts.png)

## Terraform

Terraform is an excellent option for deploying and managing cloud
infrastructure. It allows you to easily configure any number of
cloud servers as well as all the required connections and features
to ensure security, redundancy, and scalability.
Check out [the Terraform documentation to learn how it works.](https://developer.hashicorp.com/terraform/intro#how-does-terraform-work).

![Terraform](/img/explanations/terraform.png)

## Kubernetes

Kubernetes is a powerful open-source container orchestration system, initially developed by Google,
for managing containerized applications in a cluster. It aims
to provide better ways of managing related, distributed components and services
across varied infrastructure.

Kubernetes provides fine-grained controls in applications and container lifecycles,
especially in dynamic environments. Tasks such as monitoring, availability, and
deployment are simplified by the use of controllers, which are control loops that
monitor your Kubernetes cluster and make changes as needed to maintain the expected
state.
You can learn more details in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/overview/).

![Kubernetes overall structure levels](/img/explanations/kubernetes-concepts-architecture.jpg)

## Infrastructure

Below are diagrams to help show the architecture and design behind
Nebari. The diagrams are meant to show different levels and cloud specific infrastructure.

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<Tabs>
  <TabItem value="gcp" label="Google GCP" default>

![GCP Architecture Diagram](/img/explanations/architecture-diagram-gcp.png)

</TabItem>

<TabItem value="aws" label="Amazon AWS">

![AWS Architecture Diagram](/img/explanations/architecture-diagram-aws.png)

</TabItem>
<TabItem value="azure" label="Azure">

![Azure Architecture Diagram](/img/explanations/architecture-diagram-azure.png)

</TabItem>
</Tabs>



---
File: /nebari-docs/docs/docs/explanations/configuration-best-practices.mdx
---

---
title: Configuration best practices
id: config-best-practices
description: A guide to best practices for configuring Nebari
---

# Configuration best practices

This document highlights and consolidates configuration best practices for `nebari-config.yaml` that are introduced throughout the Getting Started, How To's, and Advanced Settings sections of the documentation.

## Naming conventions

When Nebari creates a new cloud resource on each one of the cloud providers, it will have to provide properties such as a name, project id, buckets, etc. for these resources.
As Nebari mainly uses the project name from config to populate those attributes, we must ensure that for such cases, the resource attribute/name meets the bound conventions for the chosen cloud provider.

To avoid any burden on the user, on keeping track of each new naming rule for all infrastructure related resources that Nebari creates, we have automated the process to ensure that all resources are named in a consistent manner.
The necessary restrictions for each cloud provider are presented below:

- Letters from A to Z (upper and lower case) and numbers are allowed;
- Special characters are NOT allowed;
- Maximum accepted length of the name string is 16 characters;
- If using AWS: names SHOULD NOT start with the string aws.

:::note
Each cloud provider has its own naming conventions which in some cases may be more restrictive or less restrictive than the ones listed above. For more information, refer to the "Initializing Nebari" section in the "How to Deploy ..." documentation for the specific cloud provider.
:::

<!--
TODO: Add domain format details
 ## Domain Format
-->

## Omitting sensitive values

If you wish to avoid storing secrets directly in the config yaml file you can instead set the values in environment variables. This substitution is triggered by setting config
values to `NEBARI_SECRET_` followed by the environment variable name.

For example, you could set the environment variables `github_client_id` and `github_client_key` and write the
following in your config file:

```yaml
security:
  authentication:
    type: GitHub
    config:
      client_id: NEBARI_SECRET_github_client_id
      client_secret: NEBARI_SECRET_github_client_key
```

## Nebari version assertion

All `nebari-config.yaml` files must contain a `nebari_version` field displaying the version of Nebari with which it’s intended to be deployed.

Typically, you can upgrade the `nebari-config.yaml` file itself by manually adding the intended version to the file and validating that the version matches the version of Nebari you are deploying with by calling `nebari --version`.

If available, you may also update the configuration using the `nebari upgrade` command. This will update image numbers, plus update `nebari_version` to match the installed version of Nebari, as well as any other bespoke changes required.

:::warning
Nebari will refuse to deploy if it doesn’t contain the same version as that of the `nebari` command.
```
Error: The schema validation of the nebari-config.yaml failed.
The following error message may be helpful in determining what went wrong:

nebari_version
  nebari_version in the config file must be equivalent to <installed version> to be
  processed by this version of Nebari (your config file version is <nebari_version>).
  Install a different version of nebari or run nebari upgrade to ensure your config
  file is compatible.
(type=value_error)
```
:::



---
File: /nebari-docs/docs/docs/explanations/index.mdx
---

import DocCardList from '@theme/DocCardList';
import {useCurrentSidebarCategory} from '@docusaurus/theme-common';

# Conceptual guides

<div align="center">
  <img
    src="/img/welcome/conceptual-guides-icon.svg"
    width="30%"
    style={{border: "none", paddingBottom: 10 + "px", boxShadow: "none"}}
  />
</div>

Big-picture explanations of higher-level Nebari concepts. Most useful for building understanding of a particular topic.

<DocCardList items={useCurrentSidebarCategory().items}/>



---
File: /nebari-docs/docs/docs/get-started/cloud-providers.mdx
---

---
title: Supported Cloud Providers
description: Guidance for selecting a deployment provider.
---

# Supported cloud providers

:::warning
While all of the cloud providers supported by Nebari offer free-tier accounts, the required resources required to deploy and run Nebari may lie outside the scope of the free-tier quotas of these accounts.
Nebari will not prompt you when these limits are exceeded so you should be aware of the limitations of your account.

If you are planning to try out Nebari by carrying out a quick trial installation, we suggest you start with a [local Nebari installation][nebari-local] to avoid any unexpected cloud charges.
:::

## Supported public cloud providers

Now that Nebari is successfully installed, we can move forward with choosing the most suitable cloud provider to help you effectively deploy and start your projects within Nebari.

<!-- TODO: add link to how-to -->

To deploy Nebari, each cloud provider requires fairly wide permissions to create all the necessary cloud resources, which are governed by the **`access keys`** generated by the provider.
Once you have chosen your preferred cloud provider, follow the steps described in their respective [How-to Guides][nebari-how-tos] to deploy Nebari on the cloud.

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<Tabs>
  <TabItem value="gcp" label="Google GCP" default>

<div class="text--center">
  <img src="/img/get-started/started-google-cloud-logo.png" width={420} />
</div>

[Google GCP](https://cloud.google.com/) (also known as Google Cloud Platform or GCP) is a provider of computing resources for developing, deploying, and operating computing services.

For detailed instructions on how to deploy Nebari on **GCP** visit the [How to deploy Nebari on GCP][nebari-gcp] section.

</TabItem>


<TabItem value="aws" label="Amazon AWS">

<div class="text--center">
  <img src="/img/get-started/started-amazon-web-services-logo.png" width={420} />
</div>

[Amazon Web Services (AWS)](https://aws.amazon.com/eks/) is a comprehensive, evolving cloud computing platform provided by Amazon that includes a mixture of infrastructure as a service (IaaS), platform as a service (PaaS) and packaged software as a service (SaaS) offerings.

For detailed instructions on how to deploy Nebari on **AWS** visit the [How to deploy Nebari on AWS][nebari-aws] section.

</TabItem>
<TabItem value="azure" label="Azure">

<div class="text--center">
  <img src="/img/get-started/started-azure-logo.png" width={420}/>
</div>

[Microsoft Azure](https://azure.microsoft.com/en-us/services/kubernetes-service/) is Microsoft's public cloud computing platform. It provides a range of cloud services, including compute, analytics, storage and networking.

For detailed instructions on how to deploy Nebari on **Azure** visit the [How to deploy Nebari on Azure][nebari-azure] section.

</TabItem>
</Tabs>

<!-- Internal links -->

[nebari-aws]: /how-tos/nebari-aws.md
[nebari-azure]: /how-tos/nebari-azure.md
[nebari-gcp]: /how-tos/nebari-gcp.md
[nebari-local]: /how-tos/nebari-local.md
[nebari-how-tos]: /how-tos/index.mdx



---
File: /nebari-docs/docs/docs/get-started/deploy.mdx
---

---
id: deploy
title: Choosing a deployment platform
---

# Choosing a deployment platform

Nebari can be deployed on any of the major public cloud providers, or on a pre-existing Kubernetes cluster.
Review the options below to determine which option best suits your needs.

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<div style={{'border-radius': '15px', 'border': '2px solid var(--ifm-color-background)', 'padding': '20px' }}>

<Tabs>
  <TabItem value="Cloud-deployment-tab" label="Cloud" default>

The cloud deployment of Nebari is considered to be the default option. It enables teams to build and maintain a cost-effective and scalable compute/data science platform in the cloud, by using an [Infrastructure as Code](https://en.wikipedia.org/wiki/Infrastructure_as_code) approach that streamlines the deployment and management of data science infrastructure.

If you are not sure which option to choose, a cloud installation is likely your best option. It is suitable for most use cases, especially if:

- You require scalable infrastructure

- You aim to have a production environment with GitOps enabled by default

- Your team does not have specific expertise within high-performance computing hardware, Kubernetes, Docker, and/or other cloud-native or scalable compute infrastructure technologies

:::note
The cloud installation is based on Kubernetes, but knowledge of Kubernetes is **NOT** required nor is in-depth knowledge about the specific provider required either.

Currently, Nebari supports [Amazon AWS][nebari-aws], [Google GCP][nebari-gcp], and [Azure][nebari-azure].
:::

</TabItem>

<TabItem value="local" label="Local deployment">

This approach is recommended for **testing and development** of Nebari’s components due to its simplicity. Choose the local mode if:

- You want to test your Kubernetes cluster
- You have available local compute setup
- You want to try out Nebari with a quick install for exploratory purposes, without setting up environment variables

You should choose another installation option, likely a [cloud install][supported-cloud-providers] if you are starting from scratch (you have no compute clusters already in place) and you desire to stand up a production instance of Nebari.

For instructions on installing and deploying Nebari Local, please visit [Deploying Nebari on a local machine](/docs/how-tos/nebari-local.md).

</TabItem>

<TabItem value="existing" label="Pre-existing Kubernetes cluster">

This approach is recommended if you are already using Kubernetes and want to deploy Nebari on your existing cluster.

For instructions on installing and deploying Nebari on an existing Kubernetes cluster, please visit [How to install and setup Nebari on an existing Kubernetes infrastructure](/docs/how-tos/nebari-kubernets.mdx).

:::note
As of now, we have only tested this functionality for AWS, but we are continuously working on expanding to other cloud providers.
:::

You should choose another installation option, likely a cloud install if you are starting from scratch (you have no compute clusters already in place) and you desire to stand up a production instance of Nebari.

</TabItem>
</Tabs>

</div>

---

## What's next?

For instructions on installing and deploying Nebari on a particular cloud provider, check out our handy how-to-guides:

- [Deploying Nebari on AWS][nebari-aws]
- [Deploying Nebari on Azure][nebari-azure]
- [Deploying Nebari on GCP][nebari-gcp]

<!-- Internal links -->

[nebari-aws]: /how-tos/nebari-aws.md
[nebari-azure]: /how-tos/nebari-azure.md
[nebari-gcp]: /how-tos/nebari-gcp.md
[nebari-local]: /how-tos/nebari-local.md
[nebari-deploy]: /get-started/deploy.mdx
[supported-cloud-providers]: /get-started/cloud-providers.mdx



---
File: /nebari-docs/docs/docs/get-started/index.mdx
---

import DocCardList from '@theme/DocCardList';
import {useCurrentSidebarCategory} from '@docusaurus/theme-common';

# Get Started with Nebari

This section describes how to install and deploy Nebari on a selection of cloud providers, and provides details on how Nebari can be configured and customized to fit a variety of deployment requirements.

<DocCardList items={useCurrentSidebarCategory().items}/>



---
File: /nebari-docs/docs/docs/get-started/installing-nebari.md
---

---
title: Installing Nebari
description: A guide to help you install Nebari for your team.
---

# Installing Nebari

This installation guide provides the basic instructions to install the Nebari deployer package CLI for the first time, and assumes you are already familiar with the [Conda](https://docs.conda.io/projects/conda/en/latest/) and [Python packaging](https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-packages) ecosystems. If you are already familiar with Nebari and would like information on advanced configuration options, skip to the [Advanced Configuration][advanced-configuration] section in this documentation.

:::note
This guide focuses on installing Nebari for **cloud usage**.

For other alternatives, visit the [Choosing a deployment platform][nebari-deploy] section for an overview of the available options and their respective installation steps.
:::

## Pre-requisites

Nebari heavily depends on [Terraform](https://www.terraform.io/) and Python. The installation of the Terraform binary is built-in within the Nebari source code, and it is automatically downloaded during the first execution. Currently, only `Linux` and `macOS` are supported. `Windows` is only supported through the “Windows Subsystem for Linux” ("WSL").

- Currently, Nebari supports `Python >= 3.8`
- For more details on Terraform and its dependencies, visit the [official Terraform documentation](https://learn.hashicorp.com/tutorials/terraform/install-cli)
- To install conda, visit the [official conda documentation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html), or if you prefer, visit the [mamba installation documentation](https://github.com/mamba-org/mamba#installation)

## How to install the Nebari Package

_See [Environment Management][environment-management] for best practices for using `conda` and `pip` to control your deployment environment._

There are currently two ways to install the Nebari CLI:

1. You can install Nebari directly from the Python Package Index (PyPI) using `pip`. For most common architectures and platforms (`Linux x86-64` and `macOS x86-64`), `pip` will download and install the most recent version available.

   ```bash
   python3 -m pip install nebari
   ```

2. Nebari is also available at [conda-forge](https://anaconda.org/conda-forge/nebari) and can be installed using the `conda` package manager by running the following command:

   ```bash
   conda install nebari -c conda-forge
   ```

   if you prefer [mamba](https://github.com/mamba-org/mamba#mamba), you can use the following command:

   ```bash
   mamba install nebari
   ```

:::note
The version of Nebari in your `nebari-config.yaml` must match your currently installed Nebari package version; otherwise, a warning will be raised when attempting to deploy. See [Upgrading Nebari][nebari-upgrade] for techniques for upgrading your Nebari CLI or safely updating your older config file to match your Nebari package version.
:::

:::note
The Nebari CLI will auto-detect and then deploy any [Nebari Extensions][nebari-extension-system] that are installed in your Python environment, and extensions once deployed cannot be uninstalled. For this reason, we recommend creating a [unique environment][environment-management] for each Nebari deployment, especially when managing multiple deployments with extensions.
:::

## Verify installation

You can verify that the Nebari package is properly installed and you can execute the client commands by running:

```bash
nebari --help
```

![A representation of the output generated when Nebari help command argument is executed, the output contains a list of the available commands and their use.](/img/get-started/nebari-help.png "Nebari's help command line output")

:::note Troubleshooting
If you are unable to successfully validate the Nebari installation above, you may want to check out our [troubleshooting guide][nebari-troubleshooting].
:::

---

## What's next?

### Nebari Init and Guided Init

Nebari Init creates and initializes your Nebari configuration. Guided init does the same but you can have a step-by-step experience while running it.

You can pass the `--help` flag to the `init` command to check all the arguments and options available for it.

```bash
nebari init --help
```

![A representation of the output generated when Nebari init help command is executed, the output contains a list of the available options and arguments and their use.](/img/get-started/nebari-init-help-2.png "Nebari's init help command line output")

:::tip
You can pass the `--guided-init` flag to the `init` command to interact with Guided Init Wizard.
:::

## Next steps?

Need more information before deploying Nebari? Check out the following sections in the documentation:

- To get more insights on the **multiple deployment methods for Nebari** - check out the [Choosing a deployment platform][nebari-deploy] guide
- To learn more about the currently **supported public cloud providers** - check out the [Supported cloud providers][supported-cloud-providers] guide

Already made your mind about deployment? Check our handy how-to-guides:

- [Deploying Nebari on AWS][nebari-aws]
- [Deploying Nebari on Azure][nebari-azure]
- [Deploying Nebari on GCP][nebari-gcp]
- [Deploying Nebari on a local cluster][nebari-local]- using [`kind`](https://kind.sigs.k8s.io/) no cloud required

<!-- Internal links -->

[advanced-configuration]: /explanations/advanced-configuration.md
[nebari-aws]: /how-tos/nebari-aws.md
[nebari-azure]: /how-tos/nebari-azure.md
[environment-management]: /how-tos/nebari-environment-management.md
[nebari-extension-system]: /how-tos/nebari-extension-system.md
[nebari-gcp]: /how-tos/nebari-gcp.md
[nebari-local]: /how-tos/nebari-local.md
[nebari-deploy]: /get-started/deploy.mdx
[nebari-troubleshooting]: /troubleshooting.mdx
[nebari-upgrade]: /how-tos/nebari-upgrade.md
[supported-cloud-providers]: /get-started/cloud-providers.mdx



---
File: /nebari-docs/docs/docs/get-started/quickstart.md
---

---
id: quickstart
title: Quickstart
description: A cheat sheet of Nebari commands for returning users.
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

This is a quick Nebari CLI reference. If you're new to Nebari, start at [Installing Nebari][install-nebari].

<div align="center">
  <img src="/img/get-started/nebari-cli-commands.png" alt="A diagram showing the different Nebari CLI commands. The first step is 'nebari init' which creates the 'nebari-config.yaml' file. The second step is 'nebari deploy' which deploys the Nebari instance on the cloud. The third step is 'nebari destroy' which destroys the deployed instance. These second and third steps run 'nebari validate' and 'nebari render' internally. 'nebari validate' verifies the 'nebari-config.yaml' file. 'nebari render' generates the 8-stage terraform deployment scripts." width="60%"/>
</div>

## Initialize

Begin by creating a new project directory:

```bash
mkdir <project-name>
cd <project-name>
```

Create the `nebari-config.yaml` file using the guided init wizard:

```bash
nebari init --guided-init
```

**Or**, if you know the initialization requirements and have set up the environment variables, you can directly run the `nebari init` command with the necessary flags.

:::caution Project naming conventions

In order to successfully deploy Nebari, there are some project naming conventions which need to be followed. Below we summarize the format requirements for the most common cloud providers:

- Letters from A to Z (upper and lower case) and numbers are allowed;
- Special characters are **NOT** allowed;
- Maximum accepted length of the name string is 16 characters;
- If using AWS names **SHOULD NOT** start with the string `aws`.

Each cloud provider has its own naming conventions which in some cases may be more restrictive or less restrictive than the ones listed above. For more information, refer to the `Initializing Nebari` section in the "How to Deploy ..." documentation for your chosen cloud provider.
:::

<Tabs>
  <TabItem value="gcp" label="GCP" default>

Download the service account key file for your Nebari project, and set the following environment variables:

```bash
export GOOGLE_CREDENTIALS="path/to/JSON/file/with/credentials"
export PROJECT_ID="Project ID"
```

Create `nebari-config.yaml` with:

```bash
nebari init gcp --project projectname \
  --domain domain \
  --auth-provider password
```

  </TabItem>
  <TabItem value="aws" label="AWS" default>

Download the file with the Access Key ID and Secret Access Key provided by your IAM role, and set the following environment variables:

```bash
export AWS_ACCESS_KEY_ID="Access Key ID"
export AWS_SECRET_ACCESS_KEY="Secret Access Key"
```

Create `nebari-config.yaml` with:

```bash
nebari init aws --project projectname \
    --domain domain \
    --auth-provider password
```

  </TabItem>

  <TabItem value="azure" label="Azure" default>

Provide authentication credentials to Nebari by setting the following environment variables:

```bash
export ARM_CLIENT_ID=""        # application (client) ID
export ARM_CLIENT_SECRET=""    # client's secret
export ARM_SUBSCRIPTION_ID=""  # Available at the `Subscription` section under the `Overview` tab
export ARM_TENANT_ID=""        # Available under `Azure Active Directories`>`Properties`>`Tenant ID`
```

Create `nebari-config.yaml` with:

```bash
nebari init azure --project projectname \
  --domain domain \
  --auth-provider password
```

  </TabItem>
</Tabs>

<!-- TODO: Add commands for HPC and local cluster. -->

## Validate (optional)

After creating the `nebari-config.yaml` file, you can customize it. The Nebari package uses Pydantic for schema validation. To ensure your customizations are valid, run:

```bash
nebari validate -c nebari-config.yaml
```

:::note
Extensions built using the [Nebari Extension System][nebari-extension-system] may extend the Nebari schema. If you are intending to use an extension and `nebari validate` returns an error `Extra inputs are not permitted`, ensure that the the correct versions of the extensions you intend to use are installed in your active Python environment.
:::

:::note
This command is automatically run when you `deploy`.
:::

## Render (optional)

You can generate the (Terraform) deployment workflow scripts with:

```bash
nebari render -c nebari-config.yaml
```

This is the actual step that loads the stage classes/models and generates physical IaC files based on your Nebari config file and installed package versions. It is not necessary to manually run a render to deploy. However, it can be useful (especially if you use a GitOps workflow with GitHub Actions or GitLab CI/CD) to review the effects of config files changes on the resulting IaC before deploying.

:::note
This command is automatically run when you `deploy`.
:::

## Deploy

<Tabs>
  <TabItem value="regular-deploy" label="Regular deploy" default>

You can deploy your Nebari instance to the cloud (selected in the previous step) with:

```bash
nebari deploy -c nebari-config.yaml
```

You may need to set up necessary DNS records (with a DNS provider of your choice) for your chosen domain to proceed if you see:

```bash
Take IP Address 12.312.312.312 and update DNS to point to "your.domain" [Press Enter when Complete]
```

  </TabItem>
  <TabItem value="auto-dns" label="Automatic DNS provision" default>

If you use Cloudflare, you can set up automatic DNS provisioning.

Create a Cloudflare API token and set the following environment variable:

```bash
export CLOUDFLARE_TOKEN="cloudflaretokenvalue"
```

Use the `--dns-auto-provision` flag with the Nebari `deploy` command:

```bash
nebari deploy -c nebari-config.yaml \
  --dns-provider cloudflare \
  --dns-auto-provision
```

  </TabItem>
</Tabs>

It can take up to 30 mins for the `deploy` command to execute.

## Destroy

To delete all your Nebari resources, while preserving the `nebari-config.yaml file`, run:

```bash
nebari destroy -c nebari-config.yaml
```

It can take up to 30 mins for the `destroy` command to complete.

:::tip
If you deployed Nebari on the cloud, verify if the relevant resources were destroyed and manually delete anything that was not destroyed.
:::

---

If you face any issues with the commands, check out the [Troubleshooting guide][nebari-troubleshooting].

<!-- internal links -->

[install-nebari]: /get-started/installing-nebari.md
[environment-management]: /how-tos/nebari-environment-management.md
[nebari-extension-system]: /how-tos/nebari-extension-system.md
[nebari-troubleshooting]: /troubleshooting.mdx



---
File: /nebari-docs/docs/docs/how-tos/access-logs-loki.md
---

---
title: Access system logs (Loki) via Grafana
description: Access common system logs on Nebari
---

# How to access system logs (Loki) via Grafana

Below is a step-by-step walkthrough of how to view JupyterHub pod logs in Grafana [Loki](https://grafana.com/docs/loki/latest/). As similar approach can be used to find other commonly accessed logs (summarized at the end of this document).

To view the Loki logs via Grafana in Nebari, you'll need to have [set up monitoring](/docs/how-tos/setup-monitoring) on your deployment.

## Getting Started

Access the Monitoring UI for your Nebari installation at \<https://\{your-nebari-domain\}/monitoring/\>.

:::note
The **Explore** functionality shown below is only available to users who have `grafana_admin` permissions. In Nebari's default configuration, only the users in the `admin` Keycloak user group will have this role. `grafana_admin` is a client role in Keycloak which can be assigned to other groups or users. See [Configure Keycloak](/docs/how-tos/configuring-keycloak#in-depth-look-at-roles-and-groups) for more information.
:::

First, click "Explore".

![Grafana Explore Page](/img/how-tos/1_grafana-explore.png)

Select Loki as the data source at the top:

![Grafana Select Loki](/img/how-tos/2_grafana-select-loki.png)

Select a Label to search. For this example, we will select `pod`:

![Grafana Select Loki](/img/how-tos/3_grafana-log-browser-pod.png)

Begin typing the name of the desired pod. In this case we are looking
for a JupyterHub pod. The exact name will vary on each deployment but it will begin with `hub-` followed by a unique identifier.

![Grafana Select Loki](/img/how-tos/4_grafana-log-search-pod.png)

Select the pod from the list of pods and then click on "Run Query":

![Grafana Select Loki](/img/how-tos/5_grafana-log-select-pod.png)

After clicking on "Run Query", you should be able to see logs for JupyterHub pod as shown below:

![Grafana Select Loki](/img/how-tos/6_grafana-view-pod-logs.png)

You can also filter by time by clicking on the time filter on top right (next to "Run query").

## Common Queries

The approach above can be used to access a wide variety of logs on Nebari. Below are some common queries.

### JupyterLab Server Logs

Each user will have their own JupyterLab instance (which may or may not be running at any given time) which will contain its own set of logs. To view the logs for a JupyterLab server, use `pod` label and begin typing the username of interest. The pod name will be `jupyter-{username}`.

### Conda-Store Logs

Conda-store runs multiple pods on the backend. The conda-store server runs continuously while conda-store-workers are only started when an environment is being built. Therefore, worker pods may not always exist. Generally, user requests and access logs will be in the server pod, while logs related to environment builds will be in a worker pod. In the list of conda-store pods shown below, you'll also see pods for `minio`, `postgresql`, and `redis`. These are used internally by conda-store are not likely to have any logs of interest.

- `nebari-conda-store-server-[id]`
- `nebari-conda-store-worker-[id]`
- `nebari-conda-store-minio-[id]`
- `nebari-conda-store-postgresql-postgresql-0`
- `nebari-conda-store-redis-master-0`

### Deployed app logs (via jhub-apps)

If you have `jhub-apps` enabled on your deployment, you can view the logs to debug deployed apps.

To see the logs from **all deployed apps**, use the label filter `container` = `notebook`. The main container in each deployed app pod is named `notebook`.

To see **logs from a specific app**, use the `pod` label and begin typing either the name of the user running the app or the app name to find the correct pod. App pods are named with the convention `jupyter-[username]--[app_name]-[pod_id]`.

## Programmatic Access to Logs

Grafana logs can be accessed programmatically from within a Jupyter Notebook or JupyterHub App running in Nebari:

- Create a Grafana Service Account and API token by following Grafana docs: https://grafana.com/docs/grafana/latest/administration/service-accounts/
- Use example code below to retrieve logs from a specific Loki Data Source UID:

```python
import requests
from datetime import datetime, timedelta
import requests
import json

NEBARI_BASE_URL = "<URL>" # Your Nebari URL e.g. https://nebari.local
GRAFANA_TOKEN = "<Token>" # e.g. "glsa_4QWcA...", See Grafana Documentation

headers = { "Authorization": f"Bearer {GRAFANA_TOKEN}" }

url = f'{NEBARI_BASE_URL}/monitoring/api/datasources/'
response = requests.get(url, headers=headers)
assert response.status_code == 200
print(response.json())
# Example output: [{'id': 2, 'uid': 'alertmanager', 'orgId': 1, 'name': 'Alertmanager', 'type': 'alertmanager',  ...

# Specify a Loki Data Source UID:
def get_loki_uid(datasources):
    for datasource in datasources:
        if datasource["name"] == "Loki":
          return datasource["uid"]

loki_datasource_uid = get_loki_uid(response.json())
print(f"Loki data source uid: {loki_datasource_uid}")
# Example output: Loki data source uid: P8E80F9AEF21F6940

# Set Up and Execute a Grafana Query:
grafana_url = f'{NEBARI_BASE_URL}/monitoring'
api_key = GRAFANA_TOKEN
loki_data_source_uid = loki_datasource_uid

# Query parameters
query = '{app="jupyterhub"}'
query_url = f'{grafana_url}/api/ds/query'

headers = { 'Authorization': f'Bearer {api_key}', 'Content-Type': 'application/json' }

# Data payload
payload = {
    'queries': [
        {
            'datasource': {'uid': loki_data_source_uid},
            'expr': query,
            'refId': 'A',
            'intervalMs': 100,
            'maxDataPoints': 1
        }
    ],
    "from":"now-5m",
   "to":"now"
}

# Send the request
response = requests.post(query_url, headers=headers, data=json.dumps(payload))

# Print the response
if response.status_code == 200:
    data = response.json()
    print(json.dumps(data["results"]["A"], indent=2))
else:
    print(f'Error: {response.status_code}, {response.text}')
```

## Additional Information

- [Understand Log Query Structure](https://grafana.com/docs/loki/latest/query/log_queries/)
- [Use the Query Editor](https://grafana.com/docs/grafana/latest/datasources/loki/query-editor/#choose-a-query-editing-mode)



---
File: /nebari-docs/docs/docs/how-tos/configure-keycloak-howto.md
---

---
id: configuring-keycloak
title: Configure Keycloak
description: |
  How to configure Keycloak, add new users, add new groups and update roles.
---

## Introduction to Keycloak

[Keycloak](https://www.keycloak.org/) is the open-source identity and access management tool that comes standard with Nebari. It is used as a centralized location to add new users, create user groups and update roles and permissions. Keycloak can be configured to work with your existing identity provider services such as GitHub, Auth0 and many others.

:::note
Although Nebari allows you to connect with GitHub and Auth0 from the `nebari-config.yaml`, any identity provider that Keycloak can connect to will work. See these [Keycloak docs](https://www.keycloak.org/docs/latest/server_admin/#_identity_broker) for more information.
:::

In Keycloak, as with many admin tools, you start with an initial login user (`root`) that has the ability to administer and create new users. The `root` user is a Keycloak-specific user. It can only be used to log in and manage the Keycloak identity management section of Nebari.

:::warning
Trying to log in to JupyterHub using this `root` user will fail. Currently, this is the only user with access to the Keycloak administration console, therefore anyone wanting to add users, new groups, etc., will need to use this user and login here: `https://{your-nebari-domain}/auth/admin/`.
:::

## Change Keycloak root password

:::warning
Once you change the root password you will not be able to [add users from the command line](#add-user-from-the-command-line)
:::

`root`'s password is generated by the `nebari init` command. If you ran this command while following the steps under [Installing Nebari][nebari-install], you should have seen something like the following in your terminal output:

```bash
Securely generated default random password=<alphanumeric-string> for Keycloak root user
```

The `init` command also saves the root user password to your `nebari-config.yaml` configuration file under the following path:

`security.keycloak.initial_root_password`

After the initial deployment, it is **highly** recommended that you change the Keycloak `root` user password as soon as possible.

1. To change the `root` user password, go to your Nebari instance's admin dashboard - e.g., something like `https://{your-nebari-domain}/auth/admin/` and log in with the root password provided.

   <p align="center">
   <img src="/img/how-tos/keycloak_master_login.png" alt="Nebari admin view - Root Login to Keycloak form" width="400"/>
   </p>

2. From there, click on the **Root** dropdown in the top right of the screen, and select **Manage account**.

   ![Keycloak root user page - manage account tab selected](/img/how-tos/keycloak_root_user_manage_account.png)

3. Under **Account Security** click **Signing In**.

   ![Keycloak root user page - account security](/img/how-tos/keycloak_root_user_account_security.png)

4. In the Password section, click the **Update** button. This will guide you through entering your existing root password, and then creating a new password.

   ![Keycloak root user page - account security, update password](/img/how-tos/keycloak_root_user_update_password.png)

:::warning
The `security.keycloak.initial_root_password` field in `nebari-config.yaml` has no effect after changing the `root` password. If you redeploy Nebari it **will not reset** the password back to the old one (or anything else that might be in the field in your YAML file). We strongly recommend you delete this field to prevent later confusion.
:::

## Adding a Nebari user

No Nebari user is created for you during deployment, you will need to add a Nebari user manually in order to [log in to JupyterHub][keycloak-login] and access the other Nebari services.

If you have chosen to use GitHub, Auth0 or any other single-sign-on, you must ensure the value you enter in Keycloak under `Username` exactly matches the username from GitHub or Auth0, respectively.

### Add user using Keycloak admin console

To add a Nebari user from the Keycloak admin console, visit `https://{your-nebari-domain}/auth/admin/` and login using the username `root`, as shown above.

Keycloak uses [realms](https://www.keycloak.org/docs/latest/server_admin/#configuring-realms) to separate their permission scopes under the Keycloak admin console. The realm for Nebari is `nebari` and all Nebari users will be part of the `nebari` realm, on the other hand, the realm for the Keycloak admin console is `master`.

:::note
The root user alone is a member of the `master` realm. For security best practices, we recommend that you minimize the number of users in the `master` realm. See the [Master realm Keycloak documentation](https://www.keycloak.org/docs/latest/server_admin/#the-master-realm) for more information.
:::

The `nebari` realm is selected by default, we strongly recommend leaving it as is.

Steps to create a new user:

1. Click **Users** along the left-hand side of the page.

2. Click the **Add user** button, and you will see the new user form:

   ![Keycloak add user tab screenshot - new user form](/img/how-tos/keycloak_add_users.png)

3. Fill out the three fields outlined above. These are **Username**, **Email**, and **Groups**. (See explanation below). Then click **save**.

   - Username: Depending on the authentication provider selected ('password', 'GitHub' or 'Auth0'), the values entered into the **Username** field will differ slightly. The following table outlines
     those differences:

     |          | Password          | GitHub            | Auth0                 |
     | -------- | ----------------- | ----------------- | --------------------- |
     | Username | _unique username_ | _GitHub username_ | _Email to login with_ |

   - Once the **Username** field is updated, add a valid email address in the **Email** field.

:::note
Although not required, users may not be able to log in to Grafana if this field isn't properly set.
:::

- Associate the user with one or more of the **Groups**. Out of the box, Nebari is deployed with the following groups: `admin`, `developer` and `analyst` (see the [Roles x Groups](#in-depth-look-at-roles-and-groups) section below for more details).

4. Click **Save**.

If you are using the password authentication provider, you will also need to define a password for the user. It's best to put the **Temporary** toggle in the **OFF** position. Otherwise, the user will be forced to change the password on first login.

:::note
If using Auth0, GitHub or any other identity provider, the password field is not required.
:::

![Keycloak add user > credentials tab screenshot - set password](/img/how-tos/keycloak_user_password.png)

### Add user from the command line

:::warning
If you [changed the initial root password for Keycloak](#change-keycloak-root-password) this method will not work.
:::

To make adding users easier for new Nebari deployments, we've created a CLI command to help you.

```shell
nebari keycloak  adduser -c nebari-config.yaml --user <username> <password>
```

This will create a new user `<username>` under the `analyst` group, with the initial password provided. Omit the password completely if you are using GitHub or Auth0. It will also add a placeholder email (i.e. `username@your-domain`) to the **Email** field.

## Login to Nebari

Your new user can now log in to Nebari, visit your provided Nebari domain URI which will take you to the login form page and follow the [How-to login in to Nebari and start a server][keycloak-login].

![Nebari - Log in to Keycloak page](/img/how-tos/nebari_login_screen.png)

## In-depth look at Roles and Groups

Groups represent a collection of users that perform similar actions and therefore require similar permissions. By default, Nebari is deployed with the following groups: `admin`, `developer`, and `analyst` (in roughly descending order of permissions and scope).

:::info
Users in a particular group will also get access to that groups shared folder. So if `user A` belongs to the `developer`, they will also have access to the `~/shared/developer` folder. This also applies to new groups that you create.
:::

Roles on the other hand represent the type or category of user. This includes access and permissions that this category of user will need to perform their regular job duties. The differences between `groups` and `roles` are subtle. Particular roles (one or many), like `conda_store_admin`, are associated with a particular group, such as `admin` and any user in this group will then assume the role of `conda_store_admin`.

:::info
These roles can be stacked. This means that if a user is in one group with role `conda_store_admin` and another group with role `conda_store_viewer`, this user ultimately has the role `conda_store_admin`.
:::

| Group        | Access to Nebari Resources                                                                                     | Roles                                                                                                                                                                     | Permissions Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| ------------ | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `analyst`    | <ul><li>Conda-Store</li><li>Jupyterhub</li><li>Argo Workflows</li><li>Grafana</li></ul>                        | <ul><li>`conda_store_developer`</li><li>`jupyterhub_developer`</li><li>`argo_viewer`</li><li>`grafana_viewer`</li></ul>                                                   | <ul><li>Default user permissions</li><li>Access to start a server instance and generate JupyterHub access token.</li><li>Read/write access to shared `analyst` folder group mount</li><li>Read access to `analyst` and write access to personal conda-store namespace</li><li>Read access to Argo-Workflows and Jupyter-Scheduler </li><li>Inherent Grafana permissions from [Grafana viewer scopes](https://grafana.com/docs/grafana/latest/administration/roles-and-permissions/#organization-roles)</li></ul> |
| `developer`  | <ul><li>Conda-Store</li><li>Dask</li><li>Jupyterhub</li><li>Argo Workflows</li><li>Grafana Developer</li></ul> | <ul><li>`conda_store_developer`</li><li>`dask_developer`</li><li>`jupyterhub_developer`</li><li>`argo_developer`</li><li>`grafana_developer`</li></ul>                    | <ul><li>All of the above access, plus...</li><li>Read access `developer` conda-store namespace</li><li>Access to create Dask clusters.</li><li>Read/write access to shared `developer` folder group mount</li><li>Read/create access to Argo-Workflows and Jupyter-Scheduler</li><li>Inherent Grafana permissions from [Grafana editor scopes](https://grafana.com/docs/grafana/latest/administration/roles-and-permissions/#organization-roles)</li></ul>                                                       |
| `admin`      | <ul><li>Conda-Store</li><li>Dask</li><li>Jupyterhub</li><li>Argo Workflows</li><li>Grafana</li></ul>           | <ul><li>`conda_store_admin`</li><li>`dask_admin`</li><li>`jupyterhub_admin`</li><li>`argo_admin`</li><li>`grafana_admin`</li></ul>                                        | <ul><li>All of the above access, plus...</li><li>Read/write access to all conda-store available namespaces/environments.</li><li>Access to Jupyterhub Admin page and can access JupyterLab users spaces</li><li>Access to Keycloak and can add remove users and groups</li><li>Inherent Grafana permissions from [Grafana administrator scopes](https://grafana.com/docs/grafana/latest/administration/roles-and-permissions/#organization-roles)</li></ul>                                                      |
| `superadmin` | <ul><li>Conda-Store</li><li>Dask</li><li>Jupyterhub</li><li>Argo Workflows</li><li>Grafana</li></ul>           | <ul><li>`conda_store_superadmin`</li><li>`dask_admin`</li><li>`jupyterhub_admin`</li><li>`argo_admin`</li><li>`realm_admin` (Keycloak) </li><li>`grafana_admin`</li></ul> | <ul><li>All of the above access, plus...</li><li>Delete (build and environment) access on conda-store</li><li>Full access to Keycloak (realm) (same as `root`)</li></ul>                                                                                                                                                                                                                                                                                                                                         |

:::info
Check [Conda-store authorization model](https://conda-store.readthedocs.io/en/latest/contributing.html#authorization-model) for more details on conda-store authorization.
:::

:::caution
The role `jupyterhub_admin` gives users elevated permissions to JupyterHub and should be applied judiciously. As mentioned in the table above, a JupyterHub admin is able to impersonate other users and view the contents of their home folder. For more details, read through the [JupyterHub documentation](https://z2jh.jupyter.org/en/stable/jupyterhub/customizing/user-management.html#admin-users).
:::

To create new groups or modify (or delete) existing groups, log in as `root` and click **Groups** on the left-hand side.

As an example, we create a new group named `conda-store-manager`. This group will have administrator access to the [Conda-Store service].

1. Click **New** in the upper-right hand corner under **Groups**.

![Keycloak groups tab screenshot - user groups view](/img/how-tos/keycloak_groups.png)

- Then, give the new group an appropriate name.

![Keycloak add group form - name field set to conda-store-manager](/img/how-tos/keycloak_new_group1.png)

2. Under **Role Mapping**, add the appropriate **Client Roles** as needed; there should be no need to update the **Realm Roles**.

![Keycloak group conda-store-manager form - role mappings tab focused with expanded client roles dropdown](/img/how-tos/keycloak_new_group2.png)

In this example, the new group only has one mapped role, `conda_store_admin`; however, it's possible to attach multiple **Client Roles** to a single group.

![Keycloak group conda-store-manager form - role mappings tab focused ](/img/how-tos/keycloak_new_group3.png)

Once complete, return to the **Users** section in the dashboard and add the relevant users to this newly created group.

<!-- internal links -->

[keycloak-login]: /docs/tutorials/login-keycloak
[nebari-install]: /docs/get-started/installing-nebari.md



---
File: /nebari-docs/docs/docs/how-tos/configure-smtp.md
---

---
id: configuring-smtp
title: Configure Outgoing SMTP in Keycloak
description: Set up your Nebari instance to send emails
---

# Introduction to Setting up Outgoing Email

Nebari deployments do not require an SMTP server for core functionality. However, certain optional features such as email validation and password resets do require outgoing emails.

Outgoing SMTP is configured at the Keycloak `nebari` realm level. These settings are currently NOT configurable in Nebari's YAML configuration `nebari-config.yaml`, but must be set manually by an administrator after the Nebari cluster is deployed.

:::note
Subsequent nebari deploy actions will NOT undo the SMTP configuration (the Terraform `keycloak_realm` resource is configured by default to ignore these changes) so it only has to be configured once.
:::

As an administrator within Keycloak web UI, navigate to the "Nebari" Realm > "Realm Settings" > "Email" tab. The general direct URL for this page is `https://[domain]/auth/admin/nebari/console/#/realms/nebari/smtp-settings`.
![Keycloak Realm - SMTP Config](/img/how-tos/nebari-smtp.png)

## Example - SMTP Using Amazon SES

Please refer to Amazon's documents for [Using SES SMTP Interface](https://docs.aws.amazon.com/ses/latest/dg/send-email-smtp.html) for the latest information on configuring SMTP.

These values generally apply:

- **Host**: email-smtp.[aws-region].amazonaws.com
- **From**: user@your-domain.com (See [AWS documentation for validating your domain](https://docs.aws.amazon.com/ses/latest/dg/creating-identities.html) - note this does NOT have to be the same domain as your Nebari instance)
- **Enable SSL**: ON
- **Enable StartTLS**: ON
- **Enable Authentication**: ON
- **Username and Password**: [Set up AWS SES Credentials for SMTP](https://docs.aws.amazon.com/ses/latest/dg/smtp-credentials.html)



---
File: /nebari-docs/docs/docs/how-tos/connect-via-ssh.mdx
---

---
id: connect-via-ssh
title: Connect to Nebari's JupyterLab via SSH or SFTP
description: Set up a remote SSH connection using jupyter-ssh
---

# Connect to Nebari's JupyterLab via SSH or SFTP

Secure Shell Protocol (SSH) is a standard protocol for communicating securely over a network connection. Secure File Transfer Protocol (SFTP) is based on SSH and transfers files to and from a remote machine. In this guide, you will learn how to connect and launch your JupyterLab user instance via SSH and transfer files using SFTP from your local machine.

Nebari includes a package called [`jupyterhub-ssh`](https://github.com/yuvipanda/jupyterhub-ssh) in all deployments to allow remote connection to their user instances. On Nebari, SSH connections are set up on port 8022 and SFTP connections on port 8023.

:::note
There is a distinction between the Nebari and JupyterHub servers, though these are easily conflated by end users.
Nebari itself is generally accessed via [kubectl](/docs/how-tos/debug-nebari#installing-kubectl)
to connect directly to the cluster (nodes or pods), while the JupyterHub server runs _on_ Nebari and controls the spawning of
user instances of JupyterLab. The approach described here uses a JupyterHub plugin, `jupyterhub-ssh`, to spawn and access users' JupyterLab instance
:::

## Prepare for the connection on the Nebari deployment

You will create a JupyterHub _token_ in order to access your server remotely. To do this, log into your Nebari deployment and navigate to the JupyterHub token page, \<your-nebari-domain.org/hub/token\>.

This page is also accessible via the Nebari Home Page:

- If your deployment has `jhub-apps` enabled, you can navigate to this page by selecting the `Token` option under your username in the upper righthand corner.
- If your deployment does not have `jhub-apps` enabled, you can click on the `Token` link on the top navigation bar.

Fill out the form for token creation:

![Form to create a new JupyterHub token](/img/how-tos/jupyterhub-token.png)

You may add a note and expiration date to the token. You can leave the scopes field blank to give the token all the permissions that your user typically has on JupyterHub (which is the service on which Nebari is running).

## Connect to Nebari via SSH

Before you begin, you'll need to have SSH installed on your local machine. From a terminal, use `ssh` to start the connection.

`ssh <your-nebari-domain.com> -l <your-username> -p 8022`

:::important
The syntax for `<your-nebari-domain.com>` is important - do not include `https://` or a trailing `/`. If you have the incorrect syntax, it will result in the error:
`Could not resolve hostname <your-nebari-domain.com>: nodename nor servname provided, or not known`
:::

You will then be prompted for your password. Enter the token as the password and press `Enter`.

If you don't already have a JupyterLab server running on your Nebari instance, the server may take a few minutes to start up. The `ssh` connection has its own timeout period, likely shorter than the time it takes to spin up a new server. Therefore, you may see the following error as the `ssh` connection times out:

`Starting your server............................................failed to start server on time!`

The good news is that Nebari is still working on spinning up your server. Retry the `ssh` connection until the connection is established once the server is up and accessible.

Once the connection is complete, you will have terminal access to your JupyterLab instance on Nebari.

## Use SFTP to transfer files to/from Nebari

By default, SFTP uses the SSH protocol to authenticate and establish a secure connection. Because of this, the same authentication token used previously in its SSH counterpart can be used here.

Utilizing SFTP will depend on how your local machine is configured and the utility you're using for SFTP. In some cases, your SSH configuration will allow SFTP to work "out of the box". Other times, special SFTP setup is required in your SSH configuration to allow for these connections.
Similar to `ssh`, from a terminal, use `sftp` to begin a new connection:

`sftp <your-nebari-domain.com> -l <your-username> -p 8023`

### Transferring Files with SFTP

Downloading Files:

- Use the `get` command to download files from the remote host:
  - `get <remote-file>` downloads the file with the same name to the local system.
  - `get <remote-file> <local-file>` downloads and saves the file with a different name.
  - To download a directory and its contents, use `get -r <some-directory>`.
  - To maintain permissions and access times, use `get -Pr <some-directory>`.

Uploading Files:

- Use the `put` command to upload files to the remote system:
  - `put <local-file>` uploads the file.
  - To upload a directory and its contents, use `put -r <some-directory>`.

## Dangerous, but useful information

:::warning
The following suggestion is not recommended unless absolutely necessary since it will artificially keep a JupyterLab instance running which may have financial repercussions.
:::

If you have a long-running task and you need to avoid the idle-culler on JupyterLab, you can use [tmux](https://github.com/tmux/tmux/wiki) to keep your server alive while your task is completing.

Short list of useful commands:

- Create a new session `tmux new -s mysession`
- Detach from session <kbd>ctrl</kbd> + <kbd>b</kbd> <kbd>d</kbd>
- List available session `tmux ls`
- Attach to a session `tmux attach -t mysession`



---
File: /nebari-docs/docs/docs/how-tos/debug-nebari.mdx
---

---
id: debug-nebari
title: Debug Nebari deployment
description: Debug your Nebari deployment
---

The beauty of Nebari is that most Dev Ops management is already handled for you.
However, there will likely come a time that you will need to debug your Nebari
instance. This debug guide contains some tips and tricks to help you in your
debug endeavors.

## Generating the `kubeconfig`

:::note
A file that is used to configure access to clusters is called a _kubeconfig_ file.
This is a generic way of referring to configuration files in the Kubernetes context.
Generally, it does not mean that there will be a file named kubeconfig present in your machine.
:::

Generating the `kubeconfig` is an essential step to get started with `kubectl`, `k9s` or
any other kubernetes management tool. The kubeconfig file contains all the necessary
information for kubectl to communicate with a Kubernetes cluster, including the API
server address, authentication credentials, and cluster details. In this section, we
will cover the process of generating a kubeconfig file and updating it to allow
kubectl to access your Nebari Kubernetes cluster.

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<div style={{'border-radius': '15px', 'border': '2px solid var(--ifm-color-background)', 'padding': '20px' }}>
<Tabs>

<TabItem value="gcp" label="Google GCP" default>

For [Google GCP](https://cloud.google.com/) (also known as Google Cloud
Platform or GCP), you'll need to have
[gcloud](https://cloud.google.com/sdk/docs/install) installed.

Next, install the `gke-gcloud-auth-plugin` component (this step will become
outdated once [GKE (Google Kubernetes Engine) version 1.26 is released](https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke)).

```bash
gcloud components install gke-gcloud-auth-plugin
```

To enable this plugin, you'll need to update `gcloud` by:

1. Set `export USE_GKE_GCLOUD_AUTH_PLUGIN=True` in `~/.bashrc` (or in
   Environment variables for Windows).

2. Run the following command:

   ```bash
   source ~/.bashrc
   ```

   For Windows environments, start a new terminal.

3. Update gcloud to the latest version.

   ```bash
   gcloud components update
   ```

Now you'll need to authenticate to GCP using:

```bash
gcloud auth login
```

Finally, you can generate the `kubeconfig` which will be used by k9s:

```bash
gcloud container clusters get-credentials <project-name>-<namespace> --region <region>
```
</TabItem>


<TabItem value="aws" label="Amazon AWS">

For [Amazon Web Services (AWS)](https://aws.amazon.com/eks/),
you'll need to have the
[AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
installed. The command to update your `kubeconfig` for AWS is:

```bash
aws eks update-kubeconfig --region <region> --name <project-name>-<namespace>
```

</TabItem>

<TabItem value="azure" label="Azure">

For [Microsoft Azure](https://azure.microsoft.com/en-us/services/kubernetes-service/),
you'll need to have the
[azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli)
installed. The command to update your `kubeconfig` for Azure is:

```bash
az ask get-credentials --resource-group <project-name>-<namespace> --name <project-name>-<namespace>
```

</TabItem>

<TabItem value="local" label="Local (Testing)">

If you deployed manually from your local machine, the configuration file
`kubeconfig` will have been autogenerated during the `nebari deploy` process.

</TabItem>

</Tabs>
</div>

## Getting started with `kubectl`

kubectl is a powerful tool that allows you to interact with Kubernetes clusters,
view and modify cluster resources, and debug issues with your deployments. In this
guide, we will cover how to install kubectl and use it to troubleshoot issues with
your Nebari deployments.

### Installing kubectl

Before proceeding, we need to make sure that kubectl is installed on your local machine.
Here are some steps to validate its presence or related documentation on how to install it:

1. Check if `kubectl` is already installed on your machine by running the following
   command:

```bash
kubectl version --client
```

2. If kubectl is not installed, you can download and install it following the instructions
   in the [kubernetes documentation](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/).
   The installation steps may vary depending on your operating system, so be sure to follow the
   instructions that apply to your system.

### Using kubectl to debug Kubernetes deployments

Once you have installed kubectl, you can use it to debug issues within your Nebari Kubernetes cluster.
kubectl provides a variety of commands and options to help you diagnose and fix issues with your deployment.
In this section, we will cover some useful `kubectl` commands to get you started.

<details>
<summary>Viewing all pods in a namespace</summary>

To view all the pods in a namespace, use the following command:

```bash
kubectl get pods -n <namespace>
```

This command lists all the pods in the specified namespace. You can use the namespace flag `(-n)`
to specify the namespace you want to view.

</details>

<details>
<summary>Viewing all pods in all namespaces</summary>

To view the logs for a specific pod, use the following command:

```bash
kubectl logs <pod-name> -n <namespace>
```

Replace `pod-name` with the name of the pod you want to view logs for, and `namespace` with
the namespace the pod is deployed in. This command displays the logs for the specified pod.

:::note
Usually Nebari is deployed into the `dev` namespace, but you can double check the assigned
value by inspecting your `nebari-config.yaml` under the `namespace` key. For more information
refer to [general configuration settings](/docs/explanations/advanced-configuration#general-configuration-settings).
:::

</details>

<details>
<summary>Viewing logs for a specific pod</summary>

```bash
kubectl exec -it <pod-name> -n <namespace> -- /bin/bash
```

Replace `pod-name` with the name of the pod you want to execute the command in, and `namespace`
with the namespace the pod is deployed in. This command opens a shell inside the specified pod,
allowing you to execute commands as if you were logged into the pod.

</details>

### Troubleshooting

:::tip
By default, `kubectl` looks for a file named config in the `$HOME/.kube` directory.
You can specify other kubeconfig files by setting the `KUBECONFIG` environment variable or
by setting the `--kubeconfig` flag. Generally, your cloud provider CLI will automatically
handle the kubeconfig directory reference for you.

For step-by-step instructions on creating and specifying kubeconfig files, see
[Configure Access to Multiple Clusters](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters).
:::

If you encounter issues connecting to your Nebari cluster using kubectl, ensure that your
kubeconfig file is set up correctly. You can refer to the previous section on [how to update your
kubeconfig](#generating-the-kubeconfig) using the cloud provider CLI.

If you are still experiencing issues, check the logs for any error messages that may provide
clues to the issue. You can also consider reaching out to the Nebari support team for assistance
by creating an issue on the Nebari issues page.

## Debugging the Kubernetes Cluster using k9s

If you need more information about your Kubernetes cluster,
[`k9s`](https://k9scli.io/) is a terminal-based UI that is extremely useful
for debugging. It simplifies navigating, observing, and managing your
applications in Kubernetes. `k9s` continuously monitors Kubernetes clusters
for changes and provides shortcut commands to interact with the observed
resources. It's a fast way to review and resolve day-to-day issues in
Kubernetes, a huge improvement to the general workflow, and a best-to-have
tool for debugging your Kubernetes cluster sessions.

### Setting up k9s for Nebari

First, follow the [installation instructions](https://k9scli.io/) on the k9s
documentation to install k9s on your local machine.

### Getting started with k9s

Launch `k9s` from a terminal on your local machine. It will look for the
configuration in the `kubeconfig` to dial the connection with the cluster.

```bash
$ k9s
```

**To view all the current processes**

Use the shortcut <kbd>0</kbd> (this may be the default).

![Image of the k9s terminal UI](/img/how-tos/k9s_UI.png)

Using `k9s` you can inspect any services launched by your cluster. For
example, if your cluster has problems with the network traffic tunnel
configuration, it may limit or block a user's access to destination resources
such as files or workers, using `k9s` you can inspect the network
configuration as well as the connection logs.

**To debug processes running on pod**

Use the arrow keys to navigate to a pod for your Nebari instance. Then,
you can investigate the pod by clicking <kbd>Enter</kbd> or by typing the
hotkey <kbd>l</kbd>.

### Advanced k9s usage

From within k9s, the port-forward keystroke, <kbd>shift</kbd> + <kbd>f</kbd>,
allows you to access and interact with your Kubernetes cluster from your
local machine. You can then use this method to investigate issues and adjust
your services locally without the need to expose them beforehand.

Let's suppose you are deploying a new service to Nebari, and would like to
quickly debug any connection related problems around it, for example a `404 Error`. With port-forwarding you can attest if the origin of the problem
lies within the service itself or with the networking aspects of the
connection, for example an Ingress route misconfiguration. This allows you to
check the service status without any extra modifications to its
structure.

### Troubleshooting

If your k9s instance fails to connect to your cluster, ensure that you have the
authentication set up properly. k9s may give you a warning similar to:
"Dial K9s Toast [X/5]".

If your k9s instance closes for unknown reasons, you can check the k9s logs.
You can specify where your log file is located by:

```bash
k9s --logFile string
```

or you can inspect the default temporary directory by looking up the `logFile`
attribute description in `k9s --help`.



---
File: /nebari-docs/docs/docs/how-tos/develop-local-packages.md
---

---
id: develop-local-packages
title: Develop Local Packages on Nebari
description: How to install a dev package from a local project
---

# How to Develop Local Packages on Nebari

As a software development platform, Nebari can be used for active package development. Since `conda-store` manages all environments, a dynamic installation of a local package requires additional care (`pip install -e .` is explicitly disallowed for various reasons, see below).

While the setup for developing local packages may require unfamiliar extra steps for new users, the total time for setup is minimal.

## Why can't I just `pip install -e` my package?

Nebari blocks the standard local installation of packages, e.g. `pip install -e .`. On your personal computer, when you `pip install` a package into a conda environment, it will be placed inside of the conda environment folder structure. However, on Nebari, the environments are managed by `conda-store` in a directory structure which is read-only to users.

For this reason, all pip installs would instead default to the user's `.local` directory. Unfortunately, _all_ environments will automatically pick up _everything_ has been installed in this directory. It is _extremely_ easy to create a situation in which all environments are broken due to a conflict. In fact, its possible to create a situation that causes a user's JupyterLab server to be unable to start. For this reason, local pip installs are prohibited in Nebari.

:::note
For more information, check out [the docs on installing pip packages](/docs/how-tos/install-pip-packages).
:::

## Installing local packages

Despite the limitations on local installations, Nebari does provide a mechanism to install a local package currently under development through the use of virtual environments.

### Building the environment

For this example, we'll walk through creating an environment called `myenv` for a user named `myusername`.

1. Use conda-store to build a basic environment that only includes Python called `myenv` in the namespace `myusername` (i.e. your own namespace).
2. Open a terminal in JupyterLab or VS Code
3. list your envs to find the exact spelling: `conda env list`
4. Activate the environment: `conda activate myusername-myenv`
5. Create a virtual environment (venv) in a folder of your choice: `python -m venv .venv_myenv`
6. Activate the venv: `source .venv_myenv/bin/activate`
7. Install your package via pip: `pip install -e .` or `pip install -r requirements.txt` (as needed)

### Usage from terminal

To use this environment from a terminal in JupyterLab or VS Code, you'll first activate the conda environment, then activate the virtual environment.

1. Activate the env: `conda activate myusername-myenv`d
2. Activate the venv: `python -m venv .venv_myenv`

### Usage from Jupyter:

To use this environment in Juptyer, you'll need to add the path to the environment to the system path since its location isn't automatically added.

1. Open a notebook with the `myusername-myenv` kernel
2. In the first notebook cell run (note that the path and python version might be different in your usecase):

```python
import sys
sys.path.append('/home/myusername/venv_myenv/lib/python3.10/site-packages/')
```

### Usage from VS Code Python extension

To use this environment to run code via the VS Code Python extension, you'll only need to point the VS Code UI to the virtual environment. This will be automatically recognized by VS Code. You will not need to add to the `sys.path` for this approach.

## Conclusion

Developing local packages on Nebari involves a few extra steps compared to working directly on your local machine. However, these additional steps are minimal and ensure a well-contained and stable environment.

To learn more about installing pip packages in general, check out the documentation on [installing pip packages from various sources](/docs/how-tos/install-pip-packages).



---
File: /nebari-docs/docs/docs/how-tos/domain-registry.md
---

---
id: domain-registry
title: Setup Nebari domain registry
description: Setting a custom DNS Domain Registry
---

# Setup Nebari domain registry

## What is a DNS?

The Domain Name System (DNS) turns domain names into IP addresses, which browsers use to load internet pages.
Every device connected to the internet has its IP address, which other devices use to locate the device.
DNS name servers are used to locate the IP address of a domain name using words or instead of direct IP addresses.

## Setting up a DNS

During deployment, Nebari will generate an [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) IP for connection with the Kubernetes cluster and all related services that Nebari runs. If not automatically handled, Nebari will request the user to generate the necessary [DNS records](https://www.cloudflare.com/en-gb/learning/dns/dns-records/) and update the domain within the newly created IP:

```bash
Take IP Address 12.312.312.312 and update DNS to point to "your.domain" [Press Enter when Complete]
```

Once the IP is generated, you will need to grab it and create the necessary records within the DNS provider of your choice. Setting a DNS record heavily depends on your provider, so an internet search for A/CNAME record for your specific provider should yield helpful results.

:::note
During the initial deployment, GCP and Azure will display an "IP" address, that you can use to set the A record.
Whereas, AWS will display a "hostname" that you can use to set the CNAME record.
:::

## Cloudflare

Nebari supports Cloudflare as a DNS provider out of the box. If you choose to use Cloudflare, first create an account, then there are two possible following options:

1. You can register your application domain name on it, using the [Cloudflare nameserver](https://developers.cloudflare.com/dns/zone-setups/full-setup/setup) (recommended).

2. You can purchase a new domain with Cloudflare.

To generate a token follow the steps below. For additional information, see the [CloudFlare docs](https://developers.cloudflare.com/api/tokens/create).

### Setting up an API token on CloudFlare

- Under Profile, select the API Tokens menu and click on Create API Token.

- On Edit zone DNS click on Use Template.![screenshot Cloudflare edit Zone DNS](/img/how-tos/cloudflare_auth_1.png).

- Configure Permissions such as the image below:![screenshot Cloudflare Permissions edit](/img/how-tos/cloudflare_permissions_2.1.1.png)

- On Account Resources set the configuration to include your desired account.![screenshot Cloudflare account resources](/img/how-tos/cloudflare_account_resources_scr.png)

- On Zone Resources set it to Include | Specific zone and your domain name.![screenshot Cloudflare zone resources](/img/how-tos/cloudflare_zone_resources.png)

- Click continue to summary.![screenshot Cloudflare summary](/img/how-tos/cloudflare_summary.png)

- Click on the Create Token button and set the token generated as an environment variable on your machine.

### Setup API token locally

Finally, set the token value as an environment variable:

```bash
 export CLOUDFLARE_TOKEN="cloudflaretokenvalue"
```

Also, add a `dns` section to the `nebari-config.yaml` file.

```yaml
dns:
  provider: cloudflare
```

## Using other DNS providers

Currently, Nebari only supports CloudFlare for [automatic DNS registration](link to automatic section below). If an alternate DNS provider is desired, change the `dns.provider` field from `cloudflare` to `none` in the `nebari-config.yaml` file.

Below are the links to detailed documentation on how to create and manage DNS records on a few providers:

- [Cloud DNS](https://cloud.google.com/dns/docs/tutorials/create-domain-tutorial) provider
- [Amazon Route 53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/domain-register.html) DNS provider
- [Azure DNS](https://docs.microsoft.com/en-us/azure/dns/dns-getstarted-portal) provider

:::note
Once your new DNS domain is created, it might take time for the records and related certificates to fully propagate.
The amount of time this takes varies for each DNS provider. Validate such information in the related documentation for your chosen provider.
:::

## Automatic DNS provision

Nebari also supports management and the creation of the DNS records for you automatically. For automatic DNS provision add `dns.auto_provision` to your Nebari config file:

```yaml
dns:
  provider: cloudflare
  auto_provision: true
```

This will set the DNS provider as Cloudflare and automatically handle the creation or updates to the Nebari domain DNS records on Cloudflare.

:::warning
The usage of `dns.auto_provision` is restricted to Cloudflare as it is the only fully integrated DNS provider that Nebari currently supports.
:::

:::warning
Earlier version of Nebari supports dns settings through `--dns-provider` and `--dns-auto-provision` flags in the `deploy` command. But this feature is removed in favor of using the `nebari-config.yaml` file.
:::

When you are done setting up the domain name, you can refer back to the [Nebari deployment documentation][nebari-deploy] and continue the remaining steps.

<!-- internal- links -->

[nebari-deploy]: /get-started/deploy.mdx



---
File: /nebari-docs/docs/docs/how-tos/fine-grained-permissions.md
---

# Fine Grained Permissions via Keycloak

Nebari provides its users (particularly admins) a way to manage roles and permissions to
various services like `jupyterhub` and `conda-store` via Keycloak. The idea is to be able to manage
roles and permissions from a central place, in this case Keycloak. An admin or anyone who has
permissions to create a role in Keycloak will create role(s) with assigned scopes (permissions)
to it and attach it to user(s) or group(s).

These roles are created and attached from keycloak's interface and scoped for a particular
client (i.e. a Nebari service such as `jupyterhub` or `conda-store`). This means the roles for a
particular service (say `jupyterhub`) should be created within the Keycloak client named
`jupyterhub`.

By default, Nebari comes with several custom clients included in a fresh deployment.
These clients facilitate various services and integrations within the Nebari ecosystem.
The predefined clients are as follows:

```yaml
clients:
  - jupyterhub
  - conda_store
  - grafana (if monitoring is enabled)
  - argo-server-sso (if argo is enabled)
  - forwardauth
```

To manage and configure these clients, you can navigate to the `Clients` tab within the
Keycloak admin console, as illustrated in the image below.

![Keycloak clients](/img/how-tos/fine_grainer_permissions_keycloak_clients.png)

This can be accessed at `<nebari-url>/auth/admin/master/console/#/realms/nebari/clients`

## Creating a Role

The process for creating a role is similar, irrespective of the service. To create a role for a
service

1. Select the appropriate client and click on "Add Role".

![Keycloak client add jupyterhub role](/img/how-tos/keycloak_jupyterhub_client.png)

2. On the "Add Role" form, write a meaningful name and description for the role. Be sure to include what this role intends to accomplish. Click "Save".

![Keycloak clients add jupyterhub role form](/img/how-tos/keycloak_jupyterhub_add_role.png)

3. Now the role has been created, but it does nothing. Let's add some permissions to it by clicking on the "Attributes" tab
   and adding scopes. The following sections will explain the `components` and `scopes` in more detail.

   ![Keycloak clients add jupyterhub role form](/img/how-tos/keycloak_add_role_attributes.png)

## Adding Role to Group(s) / User(s)

Creating a role in Keycloak has no effect on any user or group's permissions. To grant a set of permissions
to a user or group, we need to _attach_ the role to the user or group. To add a role to a user:

1. Select users on the left sidebar and enter the username in the Lookup searchbar.

   ![Keycloak clients add jupyterhub role form](/img/how-tos/keycloak_select_user.png)

2. Select that user and click on the "Role Mappings" tab.

![Keycloak clients add jupyterhub role form](/img/how-tos/keycloak_user_role_mapping_tab.png)

3. Select the Client associated with the Role being added.

![Keycloak clients add jupyterhub role form](/img/how-tos/keycloak_user_role_mapping_roles.png)

4. Select the role in the "Available Roles" and click on "Add Selected >>".

![Keycloak clients add jupyterhub role form](/img/how-tos/keycloak_user_role_mapping_add_role.png)

To attach a role to a group, follow the above steps by clicking on the groups tab and
selecting a group instead of selecting the user in the first step.

In the above section, we learned how to create a role with some attributes and attach it to a user or a group.
Now we will learn how to create scopes to grant a particular set of permissions to the user.

:::note
After the roles are assigned to a user or group in Keycloak, the user **must** logout and login back in to the service
for the roles to take in effect. For example let's say we add a set of roles for `conda-store` to the user named
"John Doe", now for the user "John Doe" to be able to avail newly granted/revoked roles, they need to login to
conda-store again (similarly for `jupyterhub` as well), after the roles are granted/revoked.
:::

### Components Attribute

We have seen in the above example the `component` attribute while creating a role. The value of this parameter
depends on the type of component in the service, we're creating a role for, currently we only have two components:

- `jupyterhub`: to create `jupyterhub` native roles in the `jupyterhub` client.
- `conda-store`: to create `conda-store` roles in the `conda_store` client

### JupyterHub Scopes

The syntax for the `scopes` attribute for a `jupyterhub` role in Keycloak in Nebari follows the native RBAC scopes syntax
for JupyterHub itself. The documentation can be found [here](https://jupyterhub.readthedocs.io/en/stable/rbac/scopes.html#scope-conventions).

As an example, scopes for allowing users to share apps in Nebari's `jhub-apps` launcher may look like this:

> `shares!user,read:users:name,read:groups:name`

The `scopes` defined above consists of three scopes:

- `shares!user`: grants permissions to share user's server
- `read:users:name`: grants permissions to read other user's names
- `read:groups:name`: grants permissions to read other groups's names

To be able to share a server to a group or a user you need to be able to read other user's or group's names and must have
permissions to be able to share your server, this is what this set of permissions implement.

### Conda Store Scopes

The scopes for roles for the `conda-store` Client are applied to the `namespace` level of `conda-store`.

Below is example of granting a user specialized permissions to `conda-store`:

> `admin!namespace=analyst,developer!namespace=nebari-git`

The `scopes` defined above consists of two scopes:

- `admin!namespace=analyst`: grants `admin` access to namespace `analyst`
- `developer!namespace=nebari-git`: grants `developer` access to namespace `nebari-git`

When attached to a user or a group, the above-mentioned permissions will be granted to the user/group.



---
File: /nebari-docs/docs/docs/how-tos/index.mdx
---

import DocCardList from '@theme/DocCardList';
import {useCurrentSidebarCategory} from '@docusaurus/theme-common';

# Nebari how-to's

<div align="center">
  <img
    src="/img/welcome/how-tos-icon.svg"
    width="30%"
    style={{border: "none", paddingBottom: 10 + "px", boxShadow: "none"}}
  />
</div>

Practical step-by-step guides to help you achieve a specific goal. Most useful when you're trying to get something done.

<DocCardList items={useCurrentSidebarCategory().items}/>



---
File: /nebari-docs/docs/docs/how-tos/install-pip-packages.md
---

---
id: install-pip-packages
title: Install pip packages
description: How to install pip packages on Nebari
---

# Installing pip packages

Nebari is supported by `conda-store` for environment management. Although all environments in Nebari are - at their core - conda environments, it is still possible to install packages via `pip`.

In this document we'll cover some common usecases. Please refer to the [pip VCS Support documentation](https://pip.pypa.io/en/stable/topics/vcs-support/) for additional usecases.

Also note that conda-store does not have the ability to pick up your SSH keys so all references must be Open Source or utilize PATs (see examples below)

:::warning
Mixing pip and conda can result in conflicts at runtime. It is best to avoid this.

If you really _must_, then continue reading.
:::

## Why can't I just use `pip install` from command line?

Nebari prohibits the standard local installation of packages, e.g. `pip install ...`. On your personal computer, when you `pip install` a package into a conda environment, it will be placed inside of the conda environment folder structure. However, on Nebari, the environments are managed by `conda-store` in a directory structure which is read-only to users.

For this reason, all pip installs would instead default to the user's `.local` directory. Unfortunately, _all_ environments will automatically pick up _everything_ has been installed in this directory. It is _extremely_ easy to create a situation in which all environments are broken due to a conflict. In fact, its possible to create a situation that causes a user's JupyterLab server to be unable to start. For this reason, local pip installs are prohibited in Nebari.

:::note
For more information on developing local packages, check out [the docs on developing packages on Nebari](/docs/how-tos/develop-local-packages).
:::

## Installing packages published on pypi.org

To install packages which are published on pypi.org, you can create an environment in conda-store as you typically would [link] but add additional section for pip. Let's use `ragna` as an example:

```yaml
channels:
- conda-forge
dependencies:
- python=3.12
- pip
- pip:
  - ragna
```

## Installing packages from an Open Source git repo

To install packages from an Open Source repository, you can use the http syntax. Its important to note that a tag or hash is required. Without it, git will attempt to install the `master` branch and will likely fail since most projects have moved away from this convention in favor of `main`.

```yaml
channels:
  - conda-forge
dependencies:
  - python=3.11
  - pip:
      - ragna @ git+https://github.com/Quansight/ragna.git@v0.2.1
```

## Installing packages from a private GitHub repository

To install a packages from a private repository on GitHub, you'll first need to [create a Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token). Please be aware that this token will be visible to anyone with access to the environment so it is best to delete the token immediately after environment creation. We suggest limiting the scope of the PAT to read-only access to the contents for a single repository.

For this example, let's suppose your GitHub user is named `username` and the PAT generated by GitHub is `github_pat_asdf`. We will assume the repo is located at https://github.com/username/reponame and we want to grab the `main` branch.

```yaml
channels:
  - conda-forge
dependencies:
  - python=3.11
  - pip
  - pip:
      - greatpackage @git+https://github_pat_asdf@github.com/username/reponame.git@main
```

## Installing packages from a private GitLab repository

To install a package from a private repository on GitLab, you'll first need to [create a PAT](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) in order to access the repo. Please be aware that this token will be visible to anyone with access to the environment so it is best to delete the token immediately after environment creation.

In this example, let's suppose your GitLab user is named `username` and the PAT generated by GitLab is `glpat-asdf`. We will assume the repo is located at https://gitlab.domain.net/teamname/projectname/reponame, the package gets installed as `greatpackage`, and we want to grab the `0.7.0` release. Your GitLab repo may have more or less team/project embedding, but this is just one example.

```yaml
channels:
  - conda-forge
dependencies:
  - python=3.11
  - pip
  - pip:
      - greatpackage @
        git+https://username:glpat-asdf@gitlab.domain.net/teamname/projectname/reponame.git@0.7.0
```

## Conclusion

Now that you've seen several examples of installing pip packages in conda-store, you can go try it out on your own packages!

To learn more about using a "dev" install of local pip package for active development, check out the documentation on [installing a development package](/docs/how-tos/develop-local-packages).



---
File: /nebari-docs/docs/docs/how-tos/jhub-app-launcher.md
---

---
id: jhub-app-launcher
title: JHub App Launcher with Nebari
description: Using JHub App Launcher with Nebari
---

# JHub App Launcher with Nebari

[JHub App Launcher](https://jhub-apps.nebari.dev/) is a generalized server launcher
for deploying web applications via JupyterHub. The App Launcher serves as the "home"
page for Nebari - providing access to various Nebari services (e.g. JupyterLab, VS Code,
conda-store) and access to deploy custom web apps.

![JHub App Launcher home screen](/img/how-tos/jhub_apps_home.png)

You can use it to create and share apps using various frameworks such as:

- Panel
- Bokeh
- Streamlit
- Plotly Dash
- Voila
- Gradio
- JupyterLab
- Any generic Python command

## Installation on Nebari

JHub App Launcher can be enabled on Nebari by adding the following in the
`nebari-config.yml`:

```yaml
jhub_apps:
  enabled: true
```

:::note
JHub App Launcher is was integrated into Nebari in version
[2023.12.1](https://github.com/nebari-dev/nebari/releases/tag/2023.12.1)
and is not enabled by default.
:::

## Overrides

This integration also supports overrides, as in configuring jhub-apps via `nebari-config.yml`.
The syntax for the same is given below:

```yaml
jhub_apps:
  enabled: true
  overrides:
    # Anything that can be customized via
    # c.JAppsConfig.<ATTRIBUTE>
    # See https://github.com/nebari-dev/jhub-apps/blob/5ed5c9d3d1eeb08a5710001fef1e63295d7cb48d/jhub_apps/config_utils.py#L5
    service_workers: 4
    blocked_frameworks:
      - jupyterlab
      - custom
```

## Usage

Documentation on how to create apps is included in the
[JHub Apps documentation](https://jhub-apps.nebari.dev/docs/category/create-apps).
Deployed apps on Nebari will utilize environments from conda-store. All apps
will need to include `jhsingle-native-proxy >= 0.8.2` in their environment along with
other framework-specific dependencies. For example, deploying a `panel` app will
require `panel` itself and additional tools for deploying `panel` apps such as
`bokeh-root-cmd >= 0.1.2`. See the documentation for specific requirements on
each framework.



---
File: /nebari-docs/docs/docs/how-tos/manual-backup.md
---

---
id: manual-backup
title: Create a manual backup
description: Perform a manual backup of a Nebari instance
---

# Manual backups

Your cloud provider may have native ways to back up your Kubernetes cluster and volumes.

This guide describes how you would manually obtain the data you need to repopulate your Nebari instance if your cluster is lost, and you wish to start it up again from the `nebari-config.yaml`
file.

There are three main locations that you need to back up:

1. The Network File System (NFS) volume where all JupyterLab workspace files are stored
2. The JupyterHub database (for Dashboard configuration)
3. The Keycloak user/group database

## Network file system

This amounts to:

- Tarballing the /home directory
- Saving to object storage \[s3, google cloud storage, etc\]
- Downloading and untaring to new cluster

This specific guide shows how to do this on an AWS cluster and upload to AWS S3.

### Pre-requisites

- [Install kubectl and update your kubeconfig](/docs/how-tos/debug-nebari#getting-started-with-kubectl)
- [Install AWS command-line tool](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)

### Pod deployment

With `kubectl` configured, the next step will be to deploy the pod that allows you to access the cluster files. First, save the following pod specification to a file named
`pod.yaml`:

```yaml
kind: Pod
apiVersion: v1
metadata:
  name: volume-debugger-ubuntu
  namespace: dev
spec:
  volumes:
    - name: volume-to-debug-ubuntu
      persistentVolumeClaim:
        claimName: "jupyterhub-dev-share"
  containers:
    - name: debugger
      image: ubuntu
      command: ["sleep", "36000"]
      volumeMounts:
        - mountPath: "/data"
          name: volume-to-debug-ubuntu
```

> Note in QHub versions before v0.4 replace `claimName: "jupyterhub-dev-share"` with `claimName: "nfs-mount-dev-share"` above.

Once the file `pod.yml` has been created, run the following command:

```shell
kubectl apply -f pod.yaml -n dev
```

If you have a namespace other than the default dev, replace `dev` with your namespace when running `kubectl`. To get a shell to this running pod, run:

```shell
kubectl exec -n dev --stdin --tty volume-debugger-ubuntu -- /bin/bash
```

Again replacing the `dev` namespace as needed.

### Installations

You must install several `apt` packages, as the pod spun up is a basic pod. Use the following command:

```shell
apt update
apt install curl -y
apt install unzip -y
```

For AWS, you need to install the CLI (see CLI instructions for Google and Azure below):

```shell
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
./aws/install
aws configure
```

The last line in the command above prompts for your AWS public/private key and default region. Paste each of these and press enter. To ignore and skip the output, press enter.

### Backups

To back up the file system, run:

```shell
cd /data
tar -cvf <custom_name>.tar .
```

The preferred naming scheme includes a year-month-day, example `2021-04-23_home_backup.tar`. You can utilize multi-backups through this step. This step takes several minutes
depending on the size of the home directories.

### Upload to object storage

Once this is complete, upload the tar file to S3 using the AWS command-line tool:

```shell
aws s3 cp 2021-04-23.tar s3://<your_bucket_name>/backups/2021-04-23.tar
```

Replacing `your_bucket_name` with a bucket you have created. If you don't have an existing bucket, check out the [instructions for creating a new bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html)

### Download from object storage and decompress

Now that the data backed up, perform the same steps preceding for the new cluster. This includes:

- Configuring kubectl for the new cluster.
- Creating a pod on the new cluster and getting shell access into it.
- Installing the apt packages.
- Configuring AWS.

Once AWS gets configured on the new pod, you can then download the backup with:

```shell
cd /data
aws s3 cp s3://<your_bucket_name>/backups/2021-04-23.tar .
```

The last step is to extract the contents of the tarball:

```shell
cd /data
tar -xvf 2021-04-23.tar
```

The file permissions for the default tar is same as the original files.

> **Important: If upgrading from 0.3.14 or earlier to 0.4 or later**
>
> QHub v0.4: If restoring your NFS as part of the upgrade you must also run some extra commands, immediately after extracting from the tar file.
>
> Previous versions contained the `shared` folder within `home`. From `0.4.0` both `shared` and `home` directories are at the same level with respect to the QHub filesystem:
>
> ```shell
> cd /data
> cp -r home/shared/* shared/
> rm -rf home/shared/
> ```
>
> And then:
>
> ```shell
> chown -R 1000:100 /data/home/*
> chown -R 1000:100 /data/shared/*
> ```
>
> From QHUb v0.4. all users will have the same `uid`.

### Google Cloud

To do a backup on Google Cloud provider, install the [gsutil](https://cloud.google.com/storage/docs/gsutil_install) CLI instead of the AWS CLI. Otherwise, the instructions are the same as
for AWS above, other than when working with S3. Here are the commands to access Google Spaces instead of S3 for copy/download of the backup:

```shell
cd /data
gsutil cp 2021-04-23.tar gs://<your_bucket_name>/backups/2021-04-23.tar

cd /data
gsutil cp gs://<your_bucket_name>/backups/2021-04-23.tar .
```

### Azure

To do a backup on Azure, first install [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=script#install-or-update-azure-cli) and [azcopy](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10?tabs=dnf#obtain-a-static-download-link). You must also have setup a storage container with blob storage. Otherwise, the instructions are the same as for AWS above, other than when working with S3. Here are the commands to access Azure blob storage instead of S3 for copy/download of the backup.

#### Do the backup

```bash
az login # --use-device-code if web browser not available

cd /data
# Tell AZCOPY to use same auth as CLI
export AZCOPY_AUTO_LOGIN_TYPE=AZCLI
# copy the tar backup file to blob storage
azcopy copy 2021-04-23.tar "https://[account].blob.core.windows.net/[container]/nebari-backups/2021-04-23.tar"
```

#### Do the restore

```bash
az login # --use-device-code if web browser not available

cd /data
# Tell AZCOPY to use same auth as CLI
export AZCOPY_AUTO_LOGIN_TYPE=AZCLI
# restore the backup file from blob storage
azcopy copy "https://[account].blob.core.windows.net/[container]/nebari-backups/2021-04-23.tar" "./2021-04-23.tar"
```

## JupyterHub Database

The JupyterHub database will mostly be recreated whenever you start a new cluster, but should be backed up to save Dashboard configurations.

You want to do something very similar to the NFS backup, above - this time you need to back up one file located in the `PersistentVolume` `hub-db-dir`.

First, you might think you can make a new `pod.yaml` file, this time specifying `claimName: "hub-db-dir"` instead of
`claimName: "jupyterhub-dev-share"`. However, `hub-db-dir`
is 'Read Write Once' - the 'Once' meaning it can only be mounted to one pod at a time
but the JupyterHub pod will already have this mounted! So the same approach will not work here.

Instead of mounting to a new 'debugger pod' you have to access the JupyterHub pod directly using the `kubectl` CLI.

Look up the JupyterHub pod:

```bash
kubectl get pods -n dev
```

It will be something like `hub-765c9488d6-8z4nj`.

Get a shell into that pod:

```bash
kubectl exec -n dev --stdin --tty hub-765c9488d6-8z4nj -- /bin/bash
```

There is no need to TAR anything up since the only file required to be backed up is `/srv/jupyterhub/jupyterhub.sqlite`.

### Backing up JupyterHub DB

Now you need to upload the file to S3. You might want to [install the AWS CLI tool](#installations) as we did before, however, as the Hub container is a rather restricted
environment the recommended approach is to upload files to AWS S3 buckets using curl.

For more details please refer to the [using curl to access AWS S3 buckets] documentation.

### Restoring JupyterHub DB

You will need to overwrite the file `/srv/jupyterhub/jupyterhub.sqlite` based on the version backed up to S3.

You should restart the pod:

```bash
kubectl delete -n dev pod hub-765c9488d6-8z4nj
```

As for uploads, [you may need to use curl to download items from an AWS S3 bucket]

## Keycloak user/group database

Nebari provides a script to export the important user/group database. Your new Nebari cluster will recreate a lot of Keycloak config (including new Keycloak clients which will
have new secrets), so only the high-level Group and User info is exported.

If you have a heavily customized Keycloak configuration, some details may be omitted in this export.

### Export Keycloak

The export script is at [`nebari/scripts/keycloak-export.py`](https://github.com/nebari-dev/nebari/blob/main/scripts/keycloak-export.py).

Locate your `nebari-config.yaml` file, for example by checking out of your Git repo for you Nebari. Activate a virtual environment with the `nebari` Python package installed.

This assumes that the password visible in the `nebari-config.yaml` file under the `security.keycloak.initial_root_password` field is still valid for the root user.

If not, first set the `KEYCLOAK_ADMIN_PASSWORD` environment variable to the new value.

Run the following to create the export file:

```shell
python nebari/scripts/keycloak-export.py -c nebari-config.yaml > exported-keycloak.json
```

You may wish to upload the Keycloak export to the same S3 location where you uploaded the TAR file in the NFS section.

### Import Keycloak

To re-import your users and groups, [login to the /auth/ URL] using the root username and password.

Under 'Manage' on the left-hand side, click 'Import'. Locate the `exported-keycloak.json` file and select it. Then click the 'Import' button.

All users and groups should now be present in Keycloak. Note that the passwords will not have been restored, so you may need to be reset them after this step.



---
File: /nebari-docs/docs/docs/how-tos/nebari-aws.md
---

---
id: nebari-aws
title: Deploy Nebari on AWS
description: A basic overview of how to deploy Nebari on AWS
---

## Introduction

This guide is to help first-time users set up an Amazon Web Services (AWS) account specifically for the purpose of using and deploying Nebari at a production scale. In this guide
we will walk you through the following steps:

- [Introduction](#introduction)
- [Sign up for Amazon Web Services](#sign-up-for-amazon-web-services)
- [Authentication](#authentication)
- [Initializing Nebari](#initializing-nebari)
- [Deploying Nebari](#deploying-nebari)
- [Destroying Nebari](#destroying-nebari)

If you are already familiar to AWS services, feel free to skip this first step and jump straight to the [Nebari authentication](#authentication) section of this guide.

## Sign up for Amazon Web Services

This documentation assumes that you are already familiar with [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) and
that you have prior knowledge regarding [AWS billing and cost usage](https://aws.amazon.com/eks/pricing/) for Kubernetes related services.

If you are new to AWS, we advise you to first [sign up for a free account](https://aws.amazon.com/free/free-tier/) to get a better understanding of the platform and its features.
Please refer to [Amazon VPC (Virtual Private Cloud)](https://aws.amazon.com/vpc/?nc1=h_ls) and
[Amazon EKS Prerequisites](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html#eks-prereqs) for more information on account types and prerequisites for
managing Kubernetes clusters.

<!-- TODO: add link to conceptual guide -->

For a more detailed cost estimate, please also refer to our Conceptual guides for more information regarding the basic infrastructure provided by Nebari.

:::warning
A Nebari deployment on AWS will **NOT** fall into `free tier` usage as some of its inner components will lead to [additional charges](https://aws.amazon.com/eks/pricing/). Therefore, we recommend that you check [AWS pricing documentation](https://aws.amazon.com/ec2/pricing/) or contact your
cloud administrator for more information. If you provision resources outside the free tier, you may be charged. We're not responsible for any charges you may incur if this
happens.
:::

## Authentication

In order for Nebari to make requests against the AWS API and create its infrastructure, an authentication method with the appropriate permissions will be required. The best way
to do this is using an [IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) with suitable permissions for your AWS account and Elastic Kubernetes Service (EKS).

As a [best practice](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#lock-away-credentials), do not use the AWS account `root` user for any task where it's not
required. Instead, create a new IAM user for each person that requires administrator access. Then make those users administrators by placing them into an "Administrators" user
group, to which you attach the `AdministratorAccess` managed policy.

If you are using an already existing IAM user, please refer to
[Managing access keys for IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) for detailed information on how to
manage your IAM user's access keys.

If it's your first time creating a new IAM user, please refer to
[Creating your first IAM admin user and user group](https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html) for more information. Otherwise, see
[Creating an IAM user in your AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) for more information on how to create an IAM user.

Follow these steps to set up your access keys and user accounts:

1. Go to [IAM in your AWS Console](https://console.aws.amazon.com/iam/home), then **Users**, and **Add Users**;
2. Give the user a name, and tick **Access Key - Programmatic access**
3. Click **Next**

   ![Account setup steps for setting your first IAM user on AWS, the image contains an input for creating your username and two item boxes for selecting the type of credential needed for this account](/img/how-tos/how-tos-aws-new-iam-user.png "Creating your IAM user account")

4. Select **Attach existing policies directly**, then select `AdministratorAccess` from the list of policies. For more information, please refer to
   [Policies and permissions in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html);
5. Then proceed with the new user creation setup.

Upon generation, the IAM role will provide a public `Access Key ID` and `Secret Access Key`. Download this file for reference later.

:::warning
The **Secret Access Key** provides access to your AWS user account. It should be treated like any other secret credentials. In particular, it should _never_ be
checked into source control.
:::

By default, Nebari will try to use the credentials associated with the current AWS infrastructure/environment for authentication. Please keep in mind that Nebari will only use
these credentials to create the first roles and stricter permissions for Nebari's internal components. Refer to \[Conceptual guides\] for more information on how Nebari's
components are secured.

Provide authentication credentials to Nebari by setting the following environment variables:

```bash
export AWS_ACCESS_KEY_ID="Access Key ID"
export AWS_SECRET_ACCESS_KEY="Secret Access Key"
```

:::tip
These environment variables will apply only to your current shell session. If you want the variables to apply to future shell sessions also, set the variables in your shell
startup file (for example, for example in the `~/.bashrc` or `~/.profile` for the bash shell). You can also opt for [`direnv`](https://direnv.net/) as a shell extension for managing your environment variables.
:::

:::note
The steps in the following sections assume you have (i) completed the [Install Nebari][nebari-install] section, (ii) confirmed that Nebari is successfully
installed in your environment, (iii) opted for **AWS** as your cloud provider, and (iv) already configured the Nebari environment variables. If you had any issues during the
installation, please visit the "Get started" section of our [troubleshooting page][nebari-troubleshooting] for further guidance.
:::

## Initializing Nebari

Great, you’ve gone through the [Nebari installation](get-started/installing-nebari.md) and [authentication setup](#authentication) steps, and have ensured that all the necessary
environment variables have been properly set.

:::warning Important

In the following steps you will be asked to provide a name for your project. This name will be used to generate the name of the infrastructure components that will be created in
your AWS account. This name must comply with the following rules:

- Must be between 3-47 characters in length;
- Must begin with an alphanumeric character, and must only contain alphanumeric lowercase characters, dashes, and underscores;
- The name cannot start with `AWS` or `aws`.

Those rules are enforced by the AWS Terraform Provider and are not configurable.
:::

In this step, you'll run `nebari init` to create the `nebari-config.yaml` file.

1. In your terminal, start by creating a new project folder. For this demonstration, we will name the new folder `nebari-aws`:

   ```bash
   mkdir nebari-aws && cd nebari-aws
   ```

<!-- TODO: replace link to configuration once migrated -->

2. Executing the `nebari init --guided-init` command prompts you to respond to a set of questions, which will be used to generate the
   [`nebari-config.yaml`](https://docs.qhub.dev/en/latest/source/installation/configuration.html) file with an infrastructure based on **AWS**.

   ```bash
   nebari init --guided-init
   ```

   ![A representation of the output generated when Nebari init guided-init command is executed.](/img/how-tos/nebari-aws.png)

:::tip
If you prefer not using the `guided-init` command then you can directly run the `init` command.

Executing the command below will generate a basic config file with an infrastructure based on **AWS**.

- `projectname` will be the name of the folder/repo that will manage this Nebari deployment (it will be created).
- `domain` will be the domain endpoint for your Nebari instance.
- `auth-provider` sets your authentication provider that you plan to use inside of Keycloak, options are Github, Auth0, and password.

For this example, we'll run with project name `projectname`, endpoint domain `domain`, and with the authentication mode set to **password**. These can be updated later by directly modifying the `nebari-config.yaml`.

```bash
nebari init aws --project projectname \
   --domain domain \
   --auth-provider password
```

You will be prompted to enter values for some choices above if they are absent from the command line arguments (for example, project name and domain)
:::

:::note
Nebari can also be deployed into an [AWS GovCloud](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/whatis.html) account, you simply need to ensure you specify the appropriate AWS GovCloud region in the `amazon_web_services.region` field of your `nebari-config.yaml`.
:::

Once `nebari init` is executed, you should then be able to see the following output:

```bash
Securely generated default random password=*** for Keycloak root user stored at path=/tmp/QHUB_DEFAULT_PASSWORD
Congratulations, you have generated the all important nebari-config.yaml file 🎉

You can always make changes to your nebari-config.yaml file by editing the file directly.
If you do make changes to it you can ensure its still a valid configuration by running:

                nebari validate --config path/to/nebari-config.yaml

For reference, if the previous Guided Init answers were converted into a direct nebari init command, it would be:

               nebari init <cloud-provider> --project-name <project-name> --domain-name <domain-name> --namespace dev --auth-provider password

You can now deploy your Nebari instance with:

        nebari deploy -c nebari-config.yaml

For more information, run nebari deploy --help or check out the documentation: https://www.nebari.dev/how-tos/
```

:::tip
The main `temp` folder on a MacOS system can be found by inspecting the value of `$TMPDIR`. This folder and its files are not meant to be user-facing and will present you
with a seemingly random directory path similar to `/var/folders/xx/xxxxx/T`
:::

You can see that Nebari is generating a random password for the root user of Keycloak. This password is stored in a temporary file and will be used to authenticate to the Keycloak
server once Nebari's infrastructure is fully deployed, to create the first user accounts for administrator(s).

The Nebari initialization scripts create a `nebari-config.yaml` file that contains a collection of default preferences and settings for your deployment.

The generated `nebari-config.yaml` is the configuration file that will determine how the cloud infrastructure and Nebari is built and deployed in the next step. Since it is a
plain text file, you can edit it manually if you are unhappy with the choices you made during initialization, or delete it and start over again by re-running `nebari init/nebari init --guided-init`.

<!-- TODO: uncomment and add link once section is added -->
<!-- For additional information about the `nebari-config.yaml` file and extra flags that allow you to configure the initialization process, see the
[Understanding the `nebari-config.yaml` file](tutorials) documentation. -->

## Deploying Nebari

To see all the options available for the deploy command, run the following command:

```bash
nebari deploy --help
```

![A representation of the output generated when nebari deploy help command is executed.](/img/how-tos/nebari-deploy-help.png)

With the `nebari-config.yaml` configuration file now created, Nebari can be deployed for the first time. Type the following command on your command line:

```bash
nebari deploy -c nebari-config.yaml
```

:::note
During deployment, Nebari will require you to set a DNS record for the domain defined during [initialize](#initializing-nebari). Follow the instructions on [How to set a DNS record for Nebari][domain-registry] for an overview of the required steps.
:::

The terminal will prompt you to press <kbd>enter</kbd> to check the authentication credentials that were added as part of the preceding `nebari init` command. Once Nebari is
authenticated, it will start its infrastructure deployment process, which will take a few minutes to complete.

If the deployment is successful, you will see the following output:

```bash
[terraform]: Nebari deployed successfully
Services:
 - argo-workflows -> https://projectname.domain/argo/
 - conda_store -> https://projectname.domain/conda-store/
 - dask_gateway -> https://projectname.domain/gateway/
 - jupyterhub -> https://projectname.domain/
 - keycloak -> https://projectname.domain/auth/
 - monitoring -> https://projectname.domain/monitoring/
Kubernetes kubeconfig located at file:///tmp/NEBARI_KUBECONFIG
Kubecloak master realm username=root *****
...
```

<!-- TODO: add link to advanced configuration -->

Congratulations! You have successfully deployed Nebari on AWS! From here, see Initial Nebari Configuration for instructions on the first steps you should take to prepare your
Nebari instance for your team's use.

## Destroying Nebari

To see all the options available for the destroy command, type the following command on your command line:

```bash
nebari destroy --help
```

![A representation of the output generated when nebari deploy help command is executed.](/img/how-tos/nebari-destroy-help.png)

Nebari also has a `destroy` command that works the same way the deploy works but instead of creating the provisioned resources it destroys it.

```bash
nebari destroy -c nebari-config.yaml
```

<!-- internal links -->

[nebari-install]: /get-started/installing-nebari.md
[nebari-troubleshooting]: /troubleshooting.mdx
[domain-registry]: /how-tos/domain-registry.md



---
File: /nebari-docs/docs/docs/how-tos/nebari-azure.md
---

---
id: nebari-azure
title: Deploy Nebari on Azure
description: A basic overview of how to deploy Nebari on Azure
---

## Introduction

This guide is to help first-time users set up an Azure account specifically for the purpose of using and deploying Nebari at a production scale. In this guide we will walk you
through the following steps:

- [Introduction](#introduction)
- [Sign up for Azure](#sign-up-for-azure)
- [Authentication](#authentication)
- [Initializing Nebari](#initializing-nebari)
- [Deploying Nebari](#deploying-nebari)
- [Destroying Nebari](#destroying-nebari)

For those already familiar to Azure subscriptions and infrastructure services, feel free to skip this first step and jump straight to the [Nebari authentication](#authentication)
section of this guide.

## Sign up for Azure

This documentation assumes that you are already familiar with [Azure accounts and subscriptions](https://docs.microsoft.com/en-us/azure/guides/developer/azure-developer-guide#understanding-accounts-subscriptions-and-billing), and that you have prior knowledge regarding [Azure billing and cost usage](https://docs.microsoft.com/en-us/azure/cost-management-billing/cost-management-billing-overview) for Kubernetes related services.

If you are new to Azure, we advise you to first [sign up for a free account](https://azure.microsoft.com/free/) to get a better understanding of the platform and its features.
Billing for Azure services is done on a per-subscription basis. For a list of the available subscription offers by type, see
[Microsoft Azure Offer Details](https://azure.microsoft.com/support/legal/offer-details/) and
[Azure Kubernetes Service](https://docs.microsoft.com/en-us/azure/ask/intro-kubernetes) for a quick overview of the Kubernetes services.

Refer to our [Conceptual guides](/docs/explanations/) for more information regarding the basic infrastructure provided by Nebari.

:::warning
A Nebari deployment on Azure will **NOT** fall into `free tier` usage as some of its inner components will lead to [additional charges](https://azure.microsoft.com/en-us/pricing/calculator/?service=kubernetes-service). Therefore, we recommend that you check [Azure pricing documentation](https://azure.microsoft.com/en-us/pricing/#product-pricing) or contact your cloud administrator for more information. If you provision resources outside the free tier, you may be charged. We're not responsible for any charges you may incur if this happens.
:::

## Authentication

In order for Nebari to make requests against the Azure API and create its infrastructure, an authentication method with the appropriate permissions will be required. The easiest
way to do this is using a [service principal](https://docs.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals#service-principal-object) with
suitable permissions for your Azure subscription.

There are three tasks necessary to create a Service Principal using the [Azure Portal](https://portal.azure.com/):

1. Create an Application in Azure Active Directory, which will create an associated Service Principal;
2. Generating a Client Secret for the Azure Active Directory Application, which Nebari will use to authenticate;
3. Grant the Service Principal access to manage resources in your Azure subscriptions

If it's your first time creating a service principal account, please refer to
[these detailed instructions to create a service principal](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/service_principal_client_secret#creating-a-service-principal-in-the-azure-portal)
for more information on how to create an Azure service principal with necessary level permissions over the Azure subscription.

By default, Nebari will try to use the credentials associated with the current Azure infrastructure/environment for authentication. Keep in mind that Nebari will only use
these credentials to create the first roles and stricter permissions for Nebari's internal components. Refer to \[Conceptual guides\] for more information on how Nebari's
components are secured.

As we've obtained the credentials for the Service Principal, provide authentication credentials to Nebari by setting the following environment variables:

```bash
export ARM_CLIENT_ID=""        # application (client) ID
export ARM_CLIENT_SECRET=""    # client's secret
export ARM_SUBSCRIPTION_ID=""  # Available at the `Subscription` section under the `Overview` tab
export ARM_TENANT_ID=""        # Available under `Azure Active Directories`>`Properties`>`Tenant ID`
```

:::tip
Having trouble finding your Subscription ID? Refer to
[Azure's official docs](https://docs.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id?tabs=portal) for more detailed information.
:::

:::tip
These environment variables will apply only to your current shell session. If you want the variables to apply to future shell sessions also, set the variables in your shell
startup file (for example, for example in the `~/.bashrc` or `~/.profile` for the bash shell). You can also opt for [`direnv`](https://direnv.net/) as a shell extension for managing your environment variables.
:::

:::note
The steps in the following sections assume you have (i) completed the [Install Nebari][nebari-install] section, (ii) confirmed that Nebari is successfully
installed in your environment, (iii) opted for **Azure** as your cloud provider, and (iv) already configured the Nebari environment variables. If you had any issues during the
installation, please visit the "Get started" section of our [troubleshooting page][nebari-troubleshooting] for further guidance.
:::

## Initializing Nebari

Great, you’ve gone through the [Nebari Installation][nebari-install] and [authentication setup](#authentication) steps, and have ensured that all the necessary
environment variables have been properly set.

:::warning Important

In the following steps you will be asked to provide a name for your project. This name will be used to generate the name of the infrastructure components that will be created in
your Azure account. This name must comply with the following rules:

- Between 3 and 16 characters long;
- Start and end with alphanumeric;
- Must use lowercase alphabets.

Those rules are enforced by Azure and are not configurable. For more information refer to [Azure's official documentation](https://learn.microsoft.com/azure/azure-resource-manager/management/resource-name-rules).
:::

In this step, you'll run `nebari init` to create the `nebari-config.yaml` file.

1. In your terminal, start by creating a new project folder. For this demonstration, we will name the new folder `nebari-azure`:

   ```bash
   mkdir nebari-azure && cd nebari-azure
   ```

2. Executing the `nebari init --guided-init` command prompts you to respond to a set of questions, which will be used to generate the
   `nebari-config.yaml` file with the Nebari cluster deployed on **Azure**.

```bash
   nebari init --guided-init
```

![A representation of the output generated when Nebari init guided-init command is executed.](/img/how-tos/nebari-azure.png)

:::tip
If you prefer not using the `guided-init` command then you can directly run the `init` command.

Executing the command below will generate a basic config file with an infrastructure based on **Azure**.

- `projectname` will be the name of the folder/repo that will manage this Nebari deployment (it will be created).
- `domain` will be the domain endpoint for your Nebari instance.
- `auth-provider` sets your authentication provider that you plan to use inside of Keycloak, options are Github, Auth0, and password.

For this example, we'll run with project name `projectname`, endpoint domain `domain`, and with the authentication mode set to **password**. These can be updated later by directly modifying the `nebari-config.yaml`.

```bash
nebari init azure --project projectname \
   --domain domain \
   --auth-provider password
```

You will be prompted to enter values for some choices above if they are absent from the command line arguments (for example, project name and domain)
:::

Once `nebari init` is executed, you should then be able to see the following output:

```bash
Securely generated default random password=*** for Keycloak root user stored at path=/tmp/QHUB_DEFAULT_PASSWORD
Congratulations, you have generated the all important nebari-config.yaml file 🎉

You can always make changes to your nebari-config.yaml file by editing the file directly.
If you do make changes to it you can ensure its still a valid configuration by running:

                nebari validate --config path/to/nebari-config.yaml

For reference, if the previous Guided Init answers were converted into a direct nebari init command, it would be:

               nebari init <cloud-provider> --project-name <project-name> --domain-name <domain-name> --namespace dev --auth-provider password

You can now deploy your Nebari instance with:

        nebari deploy -c nebari-config.yaml

For more information, run nebari deploy --help or check out the documentation: https://www.nebari.dev/how-tos/
```

:::tip
The main `temp` folder on a MacOS system can be found by inspecting the value of `$TMPDIR`. This folder and its files are not meant to be user-facing and will present you
with a seemingly random directory path similar to `/var/folders/xx/xxxxx/T`
:::

You can see that Nebari is generating a random password for the root user of Keycloak. This password is stored in a temporary file and will be used to authenticate to the Keycloak
server once Nebari's infrastructure is fully deployed, in order to create the first user accounts for administrator(s).

The Nebari initialization scripts create a `nebari-config.yaml` file that contains a collection of default preferences and settings for your deployment.

The generated `nebari-config.yaml` is the configuration file that will determine how the cloud infrastructure and Nebari is built and deployed in the next step.
Since it is a plain text file, you can edit it manually if you are unhappy with the choices you made during initialization, or delete it and start over again by re-running `nebari init`.

<!-- TODO: uncomment and add link once section is added -->
<!-- For additional information about the `nebari-config.yaml` file and extra flags that allow you to configure the initialization process, see the
[Understanding the `nebari-config.yaml` file](tutorials) documentation. -->

## Deploying Nebari

To see all the options available for the deploy command, run the following command:

```bash
nebari deploy --help
```

![A representation of the output generated when nebari deploy help command is executed.](/img/how-tos/nebari-deploy-help.png)

With the `nebari-config.yaml` configuration file now created, Nebari can be deployed for the first time. Type the following command on your command line:

```bash
nebari deploy -c nebari-config.yaml
```

:::note
During deployment, Nebari will require you to set a DNS record for the domain defined during [initialize](#initializing-nebari). Follow the instructions on [How to set a DNS record for Nebari][domain-registry] for an overview of the required steps.
:::

The terminal will prompt you to press <kbd>enter</kbd> to check the authentication credentials that were added as part of the preceding `nebari init` command. Once Nebari is
authenticated, it will start its infrastructure deployment process, which will take a few minutes to complete.

If the deployment is successful, you will see the following output:

```bash
[terraform]: Nebari deployed successfully
Services:
 - argo-workflows -> https://projectname.domain/argo/
 - conda_store -> https://projectname.domain/conda-store/
 - dask_gateway -> https://projectname.domain/gateway/
 - jupyterhub -> https://projectname.domain/
 - keycloak -> https://projectname.domain/auth/
 - monitoring -> https://projectname.domain/monitoring/
Kubernetes kubeconfig located at file:///tmp/NEBARI_KUBECONFIG
Kubecloak master realm username=root *****
...
```

<!-- TODO: Add link to advanced configuration -->

Congratulations! You have successfully deployed Nebari on Azure! From here, see Initial Nebari Configuration for instructions on the first steps you should take to prepare your
Nebari instance for your team's use.

## Destroying Nebari

To see all the options available for the destroy command, type the following command on your command line:

```bash
nebari destroy --help
```

![A representation of the output generated when nebari deploy help command is executed.](/img/how-tos/nebari-destroy-help.png)

Nebari also has a `destroy` command that works the same way the deploy works but instead of creating the provisioned resources it destroys it.

```bash
nebari destroy -c nebari-config.yaml
```

<!-- internal links -->

[nebari-install]: /get-started/installing-nebari.md
[nebari-troubleshooting]: /troubleshooting.mdx
[domain-registry]: /how-tos/domain-registry.md



---
File: /nebari-docs/docs/docs/how-tos/nebari-destroy.md
---

---
id: nebari-destroy
title: Destroy your Nebari deployment
description: how to destroy your Nebari deployment and associated resources
---

## Introduction

This is a guide on how to destroy your Nebari cluster, this includes all resources created the first time you deployed it. With your `nebari-config.yaml` configuration file, you can use a handy `destroy` command to automatically destroy resources with Nebari.

You will need to export your cloud credentials, as you would for a `nebari deploy`, before you can destroy these cloud resources.

:::warning
When working with cloud resources, it's important to always double-check that everything has been fully destroyed.

Further below, we will also take a look at how you can manually destroy resources on individual cloud providers in case the automation fails.

The Nebari team takes no responsibility for any incurred cloud costs. Please take care!
:::

## Destroying Nebari

You can specify the `nebari-config.yaml` configuration file created while deploying Nebari to destroy the resources as well. Type the following command on your command line:

:::warning
If you have any data stored on your Nebari cluster you would like to keep - either files on the file system, `conda` environments in conda-store or an exported `json` of the users and groups from Keycloak - now is the time to back those up **off of the cluster**.
:::

```bash
nebari destroy -h
```

```bash
 Usage: nebari destroy [OPTIONS]

 Destroy the Nebari cluster from your nebari-config.yaml file.

╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ *  --config          -c      TEXT  qhub configuration file path [default: None] [required]                                │
│    --output          -o      TEXT  output directory [default: ./]                                                         │
│    --disable-render                Disable auto-rendering before destroy                                                  │
│    --disable-prompt                Destroy entire Nebari cluster without confirmation request. Suggested for CI use.      │
│    --help            -h            Show this message and exit.                                                            │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

:::note
The above command will not delete your `nebar-config.yaml` or the rendered files thus a re-deployment with `nebari deploy` is possible afterwards.
:::

Once you've decided to destroy your cluster and have read through the `nebari destroy --help` command, you are you now ready.

```bash
nebari destroy -c nebari-config.yaml
```

The terminal will prompt you to press `enter` to verify that you wish to destroy your Nebari cluster.

:::info
This destruction process can take up to 30 minutes for some cloud providers.
:::

If the destruction is successful, you will see the following output:

```bash
[terraform]: Nebari properly destroyed all resources without error
```

Congratulations! You have successfully destroyed your Nebari deployment and the associated resources!

## Manually destroying the resources

### Amazon Web Services (AWS)

:::note
Destroying lingering resources on AWS can be tricky because the resources often need to be destroyed in a particular order.
:::

#### Destroy resources from the AWS console

Here, we outline how to find and destroy resources from the AWS console.

1. Sign in to your AWS account and in the search bar type `Tag Editor`

<img src="/img/how-tos/aws_tag_editor_1.png" alt="Search for Tag Editor in the AWS console" width="700"/>

2. From the `Resource Groups & Tag Editor` page, navigate to `Tag Editor` from the left most panel

<img src="/img/how-tos/aws_tag_editor_2.png" alt="Navigate to the Tag Editor in the AWS console" width="200"/>

3. Fill out the form as follows:
   - `Regions` - the region you deployed your Nebari cluster (or select `all regions`)
   - `Resource types` - select `All supported resource types`
   - `Tags`
     - select `Environment` for key (on the left)
     - enter the namespace you chose for your Nebari cluster (on the right)

<img src="/img/how-tos/aws_tag_editor_3.png" alt="Use the Tag Editor to filter for lingering Nebari resources" width="700"/>

4. From here you can filter further if needed by entering the name of your Nebari cluster.

5. At this point, select the resource you wish to destroy. This will open another tab for you to destroy that resource.

:::info
The details on how to destroy each specific resource is beyond the scope of this guide however here is a helpful tip if you get stuck in the messy web of AWS dependencies.

Try destroying resources like `EFS`, the `EC2 Load Balancer` or resources that might be connected to the `VPC` before deleting the `VPC` or any other network related resources.
:::

#### Destroy AWS resources using a dev tool script

This method of destroying lingering AWS resources requires cloning down the Nebari repo.

```bash
git clone https://github.com/Quansight/qhub.git
```

Once cloned, you can run a script which will attempt to destroy all lingering AWS resources.

```bash
python /path/to/qhub/scripts/aws-force-destroy.py -c /path/to/nebari-config.yaml
```

This will print to the terminal the resources it has destroyed. **This scripts often needs to be run multiple times before all the resources are fully destroyed.**

:::info
Even with this script, AWS on occasion will find itself in some resource dependency conflict, and you might need to destroy these resources manually from the AWS console as shown [in the corresponding section above](#destroy-resources-from-the-aws-console).
:::

### Azure

If you deployed your Nebari cluster on Azure and the `nebari destroy` command failed, you will need to destroy one resource: the resource group.

You will need [Azure's CLI `az`](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) installed and configured. Then run the following.

```bash
az group delete --resource-group "<project-name>-<namespace>"
```

Or if you'd prefer, you can destroy the resource group from the Azure portal, [follow these instructions](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal#delete-resource-groups).

### Google Cloud Platform (GCP)

If you deployed your Nebari cluster on GCP and the `nebari destroy` command failed, you will need to destroy two resources, the Kubernetes cluster (GKE) and cloud storage bucket (GCS).

You will need Google's [`gcloud`](https://cloud.google.com/sdk/gcloud) and [`gsutil`](https://cloud.google.com/storage/docs/gsutil).

To destroy the cloud storage bucket, run the following.

```bash
gsutil -m rm -r gs://<project-name>-<namespace>-terraform-state
```

And to destroy the Kubernetes cluster, run the following.

```bash
gcloud container clusters delete <project-name>-<namespace> --region <region>
```

If you wish to destroy the cloud storage bucket from the GCP console, [follow these instructions on Deleting buckets](https://cloud.google.com/storage/docs/deleting-buckets). And to destroy the Kubernetes cluster from the GCP console, [follow these instructions on Deleting a cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/deleting-a-cluster#console).

:::info
When running the destroy command on a GCP, you might encounter a message that states:

```bash
ERROR:nebari.destroy:ERROR: not all Nebari stages were destroyed properly. For cloud deployments of QHub typically only stages 01 and 02 need to succeed to properly destroy everything
```

In general, this is not a problem. This happens because the destroy command thinks it needs to destroy resources that likely no longer exist. Double check from your GCP console to be sure all resources have indeed been destroyed.

:::



---
File: /nebari-docs/docs/docs/how-tos/nebari-environment-management.md
---

---
id: nebari-environment-management
title: Setup Deployment Environment
description: Best Practices for Managing Your Python Environment
---

# Nebari Deployment Python Environment Setup

To configure and deploy the Nebari platform, you'll first need to set up your python environment.

Nebari configuration and deployment is highly dependent on the version of the Nebari CLI that you have active when you run `nebari` commands. A such, we highly recommend [installing Conda](https://docs.anaconda.com/free/anaconda/install/) for managing isolated Python environments, especially when working on more than one Nebari deployment from the same machine or using [Nebari extensions][nebari-extension-system].

Once installed, you can use [manage environments](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#managing-environments) in different ways.

This is a simple way to set up a new Conda environment named `nebari0` with a specific Nebari version and exactly one extension installed. This `nebari0` environment will store all of our packages, including those installed with `pip` because when **pip** is installed into an active conda environment, packages which are then installed with pip will not be saved in the global namespace. This especially critical when working with extensions, because `nebari` will detect and apply any extensions installed in your Python environment when it runs.

```
conda create -n nebari0
conda activate nebari0
conda install python==3.12 pip # or python version of your choice
pip install nebari==2024.1.1
# Example plugin that can also be installed
pip install nebari-plugin-self-registration==0.0.9
```

<!-- internal links -->

[nebari-extension-system]: /how-tos/nebari-extension-system.md



---
File: /nebari-docs/docs/docs/how-tos/nebari-extension-system.md
---

---
id: nebari-extension-system
title: Nebari Extension System
description: An overview of the pluggy extension system for Nebari
---

:::warning
The Extension System was added recently in release 2023.9.1. It's in active development and might be unstable.

We don't recommend using it in productions systems yet, but we'll appreciate your explorations, testing, and feedback. You can share your comments in [this GitHub Issue](https://github.com/nebari-dev/nebari/issues/1894).
:::

## Introduction

This guide is to help developers extend and modify the behavior of
Nebari. We leverage the plugin system
[pluggy](https://pluggy.readthedocs.io/en/stable/) to enable easy
extensibility. Currently, Nebari supports:

- overriding a given stages in a deployment
- adding additional stages before, after and between existing stages
- arbitrary subcommands

We maintain an [examples
repository](https://github.com/nebari-dev/nebari-plugin-examples)
which contains up to date usable examples.

## Installing a plugin

Registering a plugin should be easy for the end user. Plugins are
either installed into your existing environment e.g.

```shell
pip install my-nebari-plugin
```

Alternatively if you only want to temporarily add an extension.

:::note
`--import-plugin` does not work for loading subcommands due to
cli already being constructed before plugin imports take place.
:::

```shell
nebari --import-plugin path/to/plugin.py <command> ...
nebari --import-plugin import.module.plugin.path <command> ...
```

:::note
Currently, Nebari does **not** support un-installing a plugin.
You will need to destroy and re-deploy Nebari to remove any installed plugins.
:::

## Developing an extension

The most important step to developing a plugin is ensuring that the
setuptools entrypoint is set.

<!-- Note: FormidableLabs/prism-react-renderer, tool used by Docusaurus to generate code blocks & highlight syntax does not support TOML, so we're using YAML highlighting below -->

```yaml title="pyproject.toml"
...

[project.entry-points.nebari]
my-subcommand = "path.to.subcommand.module"

...
```

Adding this one line to your `pyproject.toml` will ensure that upon
installation of the package the Nebari plugins are registered.

### Subcommands

Nebari exposes a hook `nebari_subcommand` which exposes the
[typer](https://github.com/tiangolo/typer) CLI instance. This allows
the developer to attach any arbitrary number of subcommands.

```python
from nebari.hookspecs import hookimpl

import typer


@hookimpl
def nebari_subcommand(cli):
    @cli.command()
    def hello(
        name: str = typer.Option(
            "Nebari", help="Who to say hello to"
        )
    ):
        print(f"Hello {name}")
```

There is a dedicated working example in [nebari-plugin-examples](https://github.com/nebari-dev/nebari-plugin-examples/tree/main/examples/nebari_subcommand_hello_world).

### Stages

Nebari exposes a hook `nebari_stage` which uses the `NebariStage`
class. `NebriStage` exposes `render`, `deploy`, `destroy` as
arbitrary functions calls. See below for a complete example.

Nebari decides the order of stages based on two things the `name: str`
attribute and `priority: int` attribute. The rules are as follows:

- stages are ordered by `priority`
- stages which have the same `name` the one with highest priority
  number is chosen

```python
import contextlib
import os
from typing import Dict, Any

from nebari.hookspecs import hookimpl, NebariStage


class HelloWorldStage(NebariStage):
    name = "hello_world"
    priority = 100

    def render(self):
        return {
            "hello_world.txt": "File that says hello world"
        }

    @contextlib.contextmanager
    def deploy(self, stage_outputs: Dict[str, Dict[str, Any]]):
        print("I ran deploy")
        # set environment variables for stages that run after
        os.environ['HELLO'] = 'WORLD'
        # set output state for future stages to use
        stage_outputs[self.name] = {'hello': 'world'}
        yield
        # cleanup after deployment (rarely needed)
        os.environ.pop('HELLO')

    def check(self, stage_outputs: Dict[str, Dict[str, Any]]):
        if 'HELLO' not in os.environ:
            raise ValueError('stage did not deploy successfully since HELLO environment variable not set')

    @contextlib.contextmanager
    def destroy(self, stage_outputs: Dict[str, Dict[str, Any]], status: Dict[str, bool]):
        print('faking to destroy things for hello world stage')
        yield


@hookimpl
def nebari_stage():
    return [HelloWorldStage]
```

There is a dedicated working example in [nebari-plugin-examples](https://github.com/nebari-dev/nebari-plugin-examples/tree/main/examples/nebari_stage_hello_world).



---
File: /nebari-docs/docs/docs/how-tos/nebari-gcp.md
---

---
id: nebari-gcp
title: Deploy Nebari on GCP
description: A basic overview of how to deploy Nebari on GCP.
---

## Introduction

This guide is to help first-time users set up a Google Cloud Platform account specifically for the purpose of using and deploying Nebari at a production scale. In this guide we
will walk you through the following steps:

- [Introduction](#introduction)
- [Sign up for Google Cloud Platform](#sign-up-for-google-cloud-platform)
- [Authentication](#authentication)
- [Required GCP APIs](#required-gcp-apis)
- [Initializing Nebari](#initializing-nebari)
- [Deploying Nebari](#deploying-nebari)
- [Destroying Nebari](#destroying-nebari)

For those already familiar with Google Cloud Platform, feel free to skip this first step and jump straight to the [Nebari authentication](#authentication) section of
this guide.

:::warning important
Before version 2024.9.1, Nebari relied on users having `gcloud`, Google Cloud's CLI, installed locally on the machine they were deploying Nebari from. If you want to install an older version, make sure to [install it](https://cloud.google.com/sdk/docs/install).
:::

## Sign up for Google Cloud Platform

This documentation assumes that you are already familiar with Google Cloud Platform accounts, and that you have prior knowledge regarding GCP billing and cost usage for Kubernetes related
services.

If you are new to the Google Cloud Platform, we advise you to first [sign up for a free account](https://cloud.google.com/signup) to get a better understanding of the platform and its
features. Check the [Create a Google Account documentation](https://support.google.com/accounts/answer/27441) and
[Overview of Cloud billing concepts](https://cloud.google.com/billing/docs/concepts#billing_account) for more information on account types and cost usage. Also, please refer to
[Cluster management fee and free tier](https://cloud.google.com/kubernetes-engine/pricing#cluster_management_fee_and_free_tier) documentation for an overview of how costs are
calculated and applied to an organization's billing account.

<!-- TODO: add link to conceptual guides -->

For a more detailed cost estimate, please also refer to our \[Conceptual guides\] for more information regarding the basic infrastructure provided by Nebari.

:::note
If you are using a new GCP account, please keep in mind the [GCP quotas](https://cloud.google.com/docs/quota) of your account. Every new `free tier` account has a limited quota
for resources like `vCPU/Mem/GPUs` per region among others. Refer to [Google's free-tier limits](https://cloud.google.com/free/docs/gcp-free-tier#free-tier-usage-limits)
documentation for more information.
:::

:::warning
A Nebari deployment on GCP will **NOT** fall into `free tier` usage. Therefore, we recommend that you sign up for a paid account or contact your cloud
administrator for more information. If you provision resources outside the free tier, you may be charged. We're not responsible for any charges you may incur if this happens.
:::

The remaining steps will assume that you are logged in to a GCP account that has admin privileges for the newly created project.

## Authentication

We advise you to create a new project for Nebari deployments as this will allow you to better manage your resources and avoid any potential conflicts with other projects. This
means that any resource you create will be assigned or associated with this project. Read more about project resources at Google's
[Creating and managing projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects) documentation.

In order for Nebari to make requests against the GCP API and create its infrastructure, an authentication method with the appropriate permissions will be required. The easiest way
to do this is using a [service account](https://cloud.google.com/iam/docs/understanding-service-accounts) with suitable permissions for your GCP project and Kubernetes cluster
management.

If it's your first time creating a service account, please follow
[these detailed instructions](https://cloud.google.com/iam/docs/creating-managing-service-accounts) to create a Google Service Account with the following roles attached:

- [`roles/editor`](https://cloud.google.com/iam/docs/understanding-roles#editor)
- [`roles/resourcemanager.projectIamAdmin`](https://cloud.google.com/iam/docs/understanding-roles#resourcemanager.projectIamAdmin)
- [`roles/container.admin`](https://cloud.google.com/iam/docs/understanding-roles#container.admin)
- [`roles/storage.admin`](https://cloud.google.com/iam/docs/understanding-roles#storage.admin)

For more information about roles and permissions, see the
[Google Cloud Platform IAM documentation](https://cloud.google.com/iam/docs/choose-predefined-roles). Remember to check the active project before creating resources, especially if
you are handling multiple GCP projects.

After you create your service account, download the [service account key](https://cloud.google.com/iam/docs/reference/rest/v1/projects.serviceAccounts.keys) file following the
instructions below:

1. From the [service account key page in the Cloud Console](https://console.cloud.google.com/iam-admin/serviceaccounts) choose your recently created project;
2. Select your service account from the list;
3. Select the "Keys" tab;
4. In the drop-down menu, select "Create new key";
5. Leave the "Key Type" as `JSON`;
6. Click "Create" to create the key and save the key file to your system.

Store these credentials file in a well-known location and make sure to set yourself exclusive permissions.
You can change the file permissions by running the following command on your terminal:

```bash
chmod 600 <filename>
```

:::warning
The **service account key** file provides access to your GCP project. It should be treated like any other secret credentials. In particular, it should _never_ be
checked into source control.
:::

By default, Nebari will try to use the credentials associated with the current GCP infrastructure/environment for authentication. Please keep in mind that Nebari will only use
these credentials to create the first roles and stricter permissions for Nebari's internal components. Refer to \[Conceptual guides\] for more information on how Nebari's
components are secured.

Provide authentication credentials to Nebari by setting the following environment variables:

```bash
export GOOGLE_CREDENTIALS="path/to/JSON/file/with/credentials"
export PROJECT_ID="Project ID"
```

The **Project ID** information can be found at the Google Console homepage, under **Project Info**.

:::tip
These environment variables will apply only to your current shell session. If you want the variables to apply to future shell sessions also, set the variables in your shell
startup file (for example, for example in the `~/.bashrc` or `~/.profile` for the bash shell). You can also opt for [`direnv`](https://direnv.net/) as a shell extension for managing your environment variables.
:::

:::note
The steps in the following sections assume you have (i) completed the [Install Nebari][nebari-install] section, (ii) confirmed that Nebari is successfully
installed in your environment, (iii) opted for **GCP** as your cloud provider, and (iv) already configured the Nebari
environment variables. If you had any issues during the installation, please visit the "Get started" section of our [troubleshooting page][nebari-troubleshooting] for further
guidance.
:::

## Required GCP APIs

Deploying Nebari on GCP requires the following APIs and services enabled. So before proceeding, go to the "APIs & Services"
tab and enable the following APIs.

- Compute Engine API
- Kubernetes Engine API
- Cloud Monitoring API
- Cloud Autoscaling API
- Identity and Access Management (IAM) API
- Cloud Resource Manager API

## Initializing Nebari

Great, you’ve gone through the [Nebari Installation][nebari-install] and [authentication setup](#authentication) steps, and have ensured that all the necessary
environment variables have been properly set.

:::warning Important

In the following steps you will be asked to provide a name for your project. This name will be used to generate the name of the infrastructure components that will be created in
your GCP account. This name must comply with the following rules:

- Be 1-63 characters in length;
- Comply with [RFC 1035](https://www.ietf.org/rfc/rfc1035.txt) conventions;
- The first character must be a lowercase letter, and all the following characters must be hyphens, lowercase letters, or digits, except the last character, which cannot be a hyphen.

Those rules are enforced by GCP terraform provider and are not configurable.
:::

In this step, you'll run `nebari init` to create the `nebari-config.yaml` file.

1. In your terminal, start by creating a new project folder. For this demonstration, we will name the new folder `nebari-gcp`:

   ```bash
   mkdir nebari-gcp && cd nebari-gcp
   ```

2. Executing the `nebari init --guided-init` command prompt you to respond to a set of questions, which will be used to generate the
   `nebari-config.yaml` file with the Nebari cluster deployed on **GCP**.

```bash
   nebari init --guided-init
```

![A representation of the output generated when Nebari init guided-init command is executed.](/img/how-tos/nebari-gcp.png)

:::tip
If you prefer not using the `guided-init` command then you can directly run the `init` command.

Executing the command below will generate a basic config file with an infrastructure based on **GCP**.

- `projectname` will be the name of the folder/repo that will manage this Nebari deployment (it will be created).
- `domain` will be the domain endpoint for your Nebari instance.
- `auth-provider` sets your authentication provider that you plan to use inside of Keycloak, options are Github, Auth0, and password.

For this example, we'll run with project name `projectname`, endpoint domain `domain`, and with the authentication mode set to **password**. These can be updated later by directly modifying the `nebari-config.yaml`.

```bash
nebari init gcp --project projectname \
   --domain domain \
   --auth-provider password
```

You will be prompted to enter values for some choices above if they are absent from the command line arguments (for example, project name and domain)
:::

Once `nebari init` is executed, you should then be able to see the following output:

```bash
Securely generated default random password=*** for Keycloak root user stored at path=/tmp/QHUB_DEFAULT_PASSWORD
Congratulations, you have generated the all important nebari-config.yaml file 🎉

You can always make changes to your nebari-config.yaml file by editing the file directly.
If you do make changes to it you can ensure its still a valid configuration by running:

                nebari validate --config path/to/nebari-config.yaml

For reference, if the previous Guided Init answers were converted into a direct nebari init command, it would be:

                nebari init <cloud-provider> --project-name <project-name> --domain-name <domain-name> --namespace dev --auth-provider password

You can now deploy your Nebari instance with:

        nebari deploy -c nebari-config.yaml

For more information, run nebari deploy --help or check out the documentation: https://www.nebari.dev/how-tos/
```

:::tip
The main `temp` folder on a macOS system can be found by inspecting the value of `$TMPDIR`. This folder and its files are not meant to be user-facing and will present you
with a seemingly random directory path similar to `/var/folders/xx/xxxxx/T`
:::

You can see that Nebari is generating a random password for the root user of Keycloak. This password is stored in a temporary file and will be used to authenticate to the Keycloak
server once Nebari's infrastructure is fully deployed, in order to create the first user accounts for administrator(s).

The nebari initialization scripts create a `nebari-config.yaml` file that contains a collection of default preferences and settings for your deployment.

The generated `nebari-config.yaml` is the configuration file that will determine how the cloud infrastructure and Nebari is built and deployed in the next step. Since it is a
plain text file, you can edit it manually if you are unhappy with the choices you made during initialization, or delete it and start over again by re-running `nebari init/nebari init --guided-init`.

<!-- TODO: uncomment and add link once section is added -->
<!-- For additional information about the `nebari-config.yaml` file and extra flags that allow you to configure the initialization process, see the
[Understanding the `nebari-config.yaml` file](docs/tutorials) documentation. -->

## Deploying Nebari

To see all the options available for the deploy command, run the following command:

```bash
nebari deploy --help
```

![A representation of the output generated when nebari deploy help command is executed.](/img/how-tos/nebari-deploy-help.png)

With the `nebari-config.yaml` configuration file now created, Nebari can be deployed for the first time. Type the following command on your command line:

```bash
nebari deploy -c nebari-config.yaml
```

:::note
During deployment, Nebari will require you to set a DNS record for the domain defined during [initialize](#initializing-nebari). Follow the instructions on [How to set a DNS record for Nebari][domain-registry] for an overview of the required steps.
:::

The terminal will prompt you to press <kbd>enter</kbd> to check the authentication credentials that were added as part of the preceding `nebari init` command. Once Nebari is
authenticated, it will start its infrastructure deployment process, which will take a few minutes to complete.

If the deployment is successful, you will see the following output:

```bash
[terraform]: Nebari deployed successfully
Services:
 - argo-workflows -> https://projectname.domain/argo/
 - conda_store -> https://projectname.domain/conda-store/
 - dask_gateway -> https://projectname.domain/gateway/
 - jupyterhub -> https://projectname.domain/
 - keycloak -> https://projectname.domain/auth/
 - monitoring -> https://projectname.domain/monitoring/
Kubernetes kubeconfig located at file:///tmp/NEBARI_KUBECONFIG
Kubecloak master realm username=root *****
...
```

<!-- TODO: add link to initial config -->

Congratulations! You have successfully deployed Nebari on GCP! From here, see Initial Nebari Configuration for instructions on the first steps you should take to prepare your
Nebari instance for your team's use.

## Destroying Nebari

To see all the options available for the destroy command, type the following command on your command line:

```bash
nebari destroy --help
```

![A representation of the output generated when nebari deploy help command is executed.](/img/how-tos/nebari-destroy-help.png)

Nebari also has a `destroy` command that works the same way the deployment works but instead of creating the provisioned resources it destroys it.

```bash
nebari destroy -c nebari-config.yaml
```

<!-- internal links -->

[nebari-install]: /get-started/installing-nebari.md
[nebari-troubleshooting]: /troubleshooting.mdx
[domain-registry]: /how-tos/domain-registry.md



---
File: /nebari-docs/docs/docs/how-tos/nebari-idle-culler.md
---

---
id: idle-culling
title: Update JupyterHub idle-culling settings
description: How to update JupyterHub idle-culling settings
---

This document provides information on how to update JupyterHub idle-culling settings.

## Overview

The Nebari system includes an idle culler that automatically shuts down user notebook servers to reduce resource usage from inactive instances. There are two primary methods used for culling idle kernels and terminals.

The first method involves an internal configuration setting for the notebook server that will shut down kernels and the server itself after a certain period of inactivity. The second method involves an external JupyterHub culler ([jupyterhub-idle-culler](https://github.com/jupyterhub/jupyterhub-idle-culler)) that orchestrates the culling of idle kernels and terminals based on user usage metrics.

These two methods have different levels of granularity in terms of what they can measure and act on. The internal culler is considered more granular and can make more intelligent decisions about when to shut down kernels and servers. In contrast, the external culler is more coarse-grained and may not be as precise.

## Update JupyterHub idle-culling settings

To update the settings for managing idle kernels and terminals in Nebari/JupyterHub, modify the `nebari-config.yaml` file. The available options that can be modified are as follows:

- `terminal_cull_inactive_timeout`: The timeout (in minutes) after which a terminal is considered inactive and ready to be culled.
- `terminal_cull_interval`: The interval (in minutes) on which to check for terminals exceeding the inactive timeout value.
- `kernel_cull_idle_timeout`: The timeout (in minutes) after which an idle kernel is considered ready to be culled.
- `kernel_cull_interval`: The interval (in minutes) on which to check for idle kernels exceeding the cull timeout value.
- `kernel_cull_connected`: Whether to consider culling kernels which have one or more connections.
- `kernel_cull_busy`: Whether to consider culling kernels which are currently busy running some code.
- `server_shutdown_no_activity_timeout`: The timeout (in minutes) after which the server is shut down if there is no activity.

To update any of these options, modify the corresponding value in the `nebari-config.yaml` file and save the changes. For example, to update the `kernel_cull_idle_timeout` value to 30 minutes, modify the following lines:

```yaml
jupyterlab:
  idle_culler:
    kernel_cull_idle_timeout: 30
```

:::note
Note that once the new configurations are applied and the user pod is left idle, it should take approximately the sum of `max(cull_idle_timeout, cull_inactive_timeout)` (if both are enabled) plus `shutdown_no_activity_timeout` for the pod to be terminated. However, these configurations are approximate and may not reflect the exact time that a kernel or terminal will be terminated. The exact timing may depend on factors such as server load, network latency, and other system resources that could lead to a certain amount of variance in the actual termination time.
:::

## Default values

The idle culling mechanism is enabled by default in Nebari, and it can be configured using the `nebari-config.yaml` file. By default, JupyterHub will ping the user notebook servers every 10 minutes to check their status. Every server found to be idle for more than 30 minutes (approximately) will be culled. The default values for the internal culling mechanism are as follows:

```yaml
jupyterlab:
  idle_culler:
    kernel_cull_busy                    : false
    kernel_cull_connected               : true
    kernel_cull_idle_timeout            : 15
    kernel_cull_interval                : 5
    server_shutdown_no_activity_timeout : 15
    terminal_cull_inactive_timeout      : 15
    terminal_cull_interval              : 5
```



---
File: /nebari-docs/docs/docs/how-tos/nebari-kubernets.mdx
---

---
id: nebari-kubernetes
title: How to deploy Nebari on pre-existing infrastructure
description: Deploying Nebari on an existing Kubernetes infrastructure
---

Nebari can also be deployed on top of Kubernetes clusters. In this documentation,
we will guide you through the process of deploying Nebari into a pre-existing Kubernetes cluster.

To make it easier for you to follow along, we will outline the steps for such deployment with a
simple infrastructure example. We will use tabs to represent the different provider
steps/configurations. Let's get started!

### Evaluating the infrastructure

This is an optional stage and will only be used as part of this guided example, for setting up an initial infrastructure.

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<div style={{'border-radius': '15px', 'border': '2px solid var(--ifm-color-background)', 'padding': '20px' }}>
<Tabs className="unique-tabs">
<TabItem value="AWS">

In this example, a basic web app is already running on an EKS cluster. Here is a tutorial on [how to set
up the Guestbook web app](https://logz.io/blog/amazon-eks-cluster/), containing more details.

The existing EKS cluster has one Virtual Private Cloud (VPC) with three subnets, each
in its Availability Zone, and no node groups. There are three nodes running on a `t3.medium` EC2 instance, but unfortunately,
Nebari's general node group requires a more powerful instance type.

Before proceeding, ensure that the subnets can
"[automatically assign public IP addresses to instances launched into it](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-ip-addressing.html#subnet-public-ip)" and
that there exists an [Identity and Access Management (IAM)](https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html) role with the following permissions:

- `AmazonEKSWorkerNodePolicy`
- `AmazonEC2ContainerRegistryReadOnly`
- `AmazonEKS_CNI_Policy`

<details>
<summary>Custom CNI policy (Click to expand) </summary>

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "eksWorkerAutoscalingAll",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeLaunchTemplateVersions",
                "autoscaling:DescribeTags",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeAutoScalingGroups"
            ],
            "Resource": "*"
        },
        {
            "Sid": "eksWorkerAutoscalingOwn",
            "Effect": "Allow",
            "Action": [
                "autoscaling:UpdateAutoScalingGroup",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "autoscaling:SetDesiredCapacity"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "autoscaling:ResourceTag/k8s.io/cluster-autoscaler/enabled": [
                        "true"
                    ],
                    "autoscaling:ResourceTag/kubernetes.io/cluster/eaeeks": [
                        "owned"
                    ]
                }
            }
        }
    ]
}
```

</details>
</TabItem>
</Tabs>
</div>

### Creating node groups

Skip this step if node groups already exists.

<div style={{'border-radius': '15px', 'border': '2px solid var(--ifm-color-background)', 'padding': '20px' }}>
<Tabs className="unique-tabs">
<TabItem value="AWS">

[Follow this guide to create new node groups](https://docs.aws.amazon.com/eks/latest/userguide/create-managed-node-group.html).
Be sure to fill in the following fields carefully:

- "Node Group configuration"
  - `Name` must be either `general`, `user` or `worker`
  - `Node IAM Role` must be the IAM role described proceeding
- "Node Group compute configuration"
  - `Instance type`
    - The recommended minimum vCPU and memory for a `general` node is 8 vCPU / 32 GB RAM
    - The recommended minimum vCPU and memory for a `user` and `worker` node is 4 vCPU / 16 GB RAM
  - `Disk size`
    - The recommended minimum is 200 GB for the attached EBS (block-storage)
- "Node Group scaling configuration"
  - `Minimum size` and `Maximum size` of 1 for the `general` node group
- "Node Group subnet configuration"
  - `subnet` include all existing EKS subnets

</TabItem>
</Tabs>
</div>

### Deploying Nebari

Ensure that you are using the existing cluster's `kubectl` context.

<div style={{'border-radius': '15px', 'border': '2px solid var(--ifm-color-background)', 'padding': '20px' }}>
<Tabs className="unique-tabs">
<TabItem value="AWS">

Initialize in the usual manner:

```
python -m nebari init aws --project <project_name> --domain <domain_name> --ci-provider github-actions --auth-provider github --auth-auto-provision
```

Then update the `nebari-config.yaml` file. The important keys to update are:

- Replace `provider: aws` with `provider: existing`
- Replace `amazon_web_services` with `local`
  - And update the `node_selector` and `kube_context` appropriately

<details>
<summary>Example nebari-config.yaml (Click to expand)</summary>

```
project_name: <project_name>
provider: existing
domain: <domain_name>
certificate:
  type: self-signed
security:
  authentication:
    type: GitHub
    config:
      client_id:
      client_secret:
      oauth_callback_url: https://<domain_name>/hub/oauth_callback
...
ci_cd:
  type: github-actions
  branch: main
terraform_state:
  type: remote
namespace: dev
local:
  kube_context: arn:aws:eks:<region>:xxxxxxxxxxxx:cluster/<existing_cluster_name>
  node_selectors:
    general:
      key: eks.amazonaws.com/nodegroup
      value: general
    user:
      key: eks.amazonaws.com/nodegroup
      value: user
    worker:
      key: eks.amazonaws.com/nodegroup
      value: worker
profiles:
  jupyterlab:
  - display_name: Small Instance
    description: Stable environment with 2 cpu / 8 GB ram
    default: true
    kubespawner_override:
      cpu_limit: 2
      cpu_guarantee: 1.5
      mem_limit: 8G
      mem_guarantee: 5G
      image: quansight/nebari-jupyterlab:latest
  - display_name: Medium Instance
    description: Stable environment with 4 cpu / 16 GB ram
    kubespawner_override:
      cpu_limit: 4
      cpu_guarantee: 3
      mem_limit: 16G
      mem_guarantee: 10G
      image: quansight/nebari-jupyterlab:latest
  dask_worker:
    Small Worker:
      worker_cores_limit: 2
      worker_cores: 1.5
      worker_memory_limit: 8G
      worker_memory: 5G
      worker_threads: 2
      image: quansight/nebari-dask-worker:latest
    Medium Worker:
      worker_cores_limit: 4
      worker_cores: 3
      worker_memory_limit: 16G
      worker_memory: 10G
      worker_threads: 4
      image: quansight/nebari-dask-worker:latest
environments:
...
```

</details>

Once updated, deploy Nebari. When prompted be ready to manually update the DNS record.

- `local` or "existing" deployments fail if you pass `--dns-auto-provision` or `--disable-prompt`

```
python -m nebari deploy --config nebari-config.yaml
```

The deployment completes successfully and all the pods appear to be running and so do the pre-existing Guestbook web
app.

</TabItem>
</Tabs>
</div>



---
File: /nebari-docs/docs/docs/how-tos/nebari-local.md
---

---
id: nebari-local
title: Deploy Nebari on local machines (for testing)
description: Deploying Nebari on a local development environment with kind
---

Nebari supports a "local testing" mode that helps you develop and efficiently run Nebari on any environment, which can vary from a Virtual Machine, a laptop, or a sizeable multi-node ecosystem.

This installation mode is intended to be used as a debugging tool or, to some extent, as a preview of Nebari functionalities. Thus we do not recommend its use in a production environment.
It's important to highlight that while it's possible to test most of Nebari with this version, components that are Cloud provisioned such as VPCs, managed Kubernetes cluster, and managed container registries can't be locally tested, due to their Cloud dependencies.

To operate in this mode, Nebari uses "kind" to maintain and deploy the Kubernetes environment where its resources will be integrated with.
This deployment mode allows quicker feedback loops for development, as well as removes the requisites imposed by using a cloud Kubernetes cluster.

<!-- TODO: Add link to nebari-kubernetes page when ready -->

<!-- Note: If you are looking for installation and deployment instructions for an existing Kubernetes cluster, please visit [Deploy Nebari on an existing Kubernetes infrastructure]. -->

## What is kind?

[kind](https://kind.sigs.k8s.io/) is a tool for running local Kubernetes clusters using Docker container "nodes".
kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

The advantages of using kind are:

- kind supports multi-node (including HA) clusters
- kind supports building Kubernetes release builds from source
  - support for make / bash or docker, in addition to pre-published builds
- kind supports Linux, macOS and Windows
- kind is a [CNCF certified conformant Kubernetes installer](https://landscape.cncf.io/?selected=kind)

## Compatibility

Nebari integrates kind under the hood by using its Terraform provider and a proper `local` deployment method, which grants native OS compatibility with Linux.

:::warning
Currently, Nebari does not support local mode on Windows.
:::

To use kind, you will also need to install [docker engine](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository).

:::note
While `kubectl` is not required for kind to work, we do recommended its installation as it provides an excellent interface for interacting with the cluster resources.
To install kubectl and configure access to the Nebari cluster, refer to [Install and Set Up kubectl](/docs/how-tos/debug-nebari#getting-started-with-kubectl).
:::

## Initializing Nebari

The following steps assume you have:

- An installed version of Nebari, for any directions please visit [Install Nebari](/docs/get-started/installing-nebari) section,
- confirmed that `nebari` is successfully installed in your environment.

1. In your terminal, start by creating a new project folder. For this demonstration, we will name the new folder `nebari-local`:

   ```bash
   mkdir nebari-local && cd nebari-local
   ```

2. Executing the command below will generate a basic config file with an infrastructure based on **local**, with project name `projectname`, endpoint domain `domain`, and with the authentication mode set to **password**.

   ```bash
   nebari init local \
    --project projectname \
    --domain domain \
    --auth-provider password \
    --terraform-state=local
   ```

   Where `--terraform-state=local` is used to tell Nebari to store the Terraform state in the **local** filesystem.

:::note
Note You will be prompted to enter values for some of the choices above if they are absent from the command line arguments (for example, project name and domain)
:::

:::note
If you are deploying for testing purposes, omitting the `--domain` option is recommended to prevent issues related to setting up a DNS domain for your deployment. By default, at the end of your deployment, Nebari will return the external IP associated with the Kubernetes cluster's load balancer.
:::

Once `nebari init` is executed, you should then be able to see the following output:

```bash
Securely generated default random password=*** for Keycloak root user
stored at path=/tmp/NEBARI_DEFAULT_PASSWORD
```

:::tip
The main `temp` folder on a MacOS system can be found by inspecting the value of `$TMPDIR`.
This folder and its files are not meant to be user-facing and will present you
with a seemingly random directory path similar to `/var/folders/xx/xxxxx/T`
:::

You can see that Nebari is generating a random password for the root user of Keycloak. This password is stored in a temporary file and will be used to authenticate to the Keycloak
server once Nebari's infrastructure is fully deployed, in order to create the first user accounts for administrator(s).

The Nebari initialization scripts create a `nebari-config.yaml` file that contains a collection of default preferences and settings for your deployment.

The generated `nebari-config.yaml` is the configuration file that will determine how the cloud infrastructure and Nebari is built and deployed in the next step.
Since it is a regular text file, you can edit it manually if you are unhappy with the choices you made during initialization, or delete it and start over again by re-running `nebari init`.

:::tip
You can also use the new Guided Init wizard for a step-by-step initialization process.

```bash
nebari init --guided-init
```

:::

<!-- TODO: Add link to advances settings doc -->
<!-- For additional information about the `nebari-config.yaml` file and extra flags that allow you to configure the initialization process, see the
[Understanding the nebari-config.yaml file]() documentation. -->

### Setting up a DNS record for the domain

In order to deploy the infrastructure, we need to set up a DNS record for the `<domain>` you provided during initialization. This will be required for the infrastructure to be able to resolve the domain name. The local deployment sets the Ingress IP to `172.18.1.100` and assigns this to your domain.

To make this easier, open your `/etc/hosts` file with your favorite text editor and add the following line:

```bash
172.18.1.100 <domain>
```

:::tip
If you are a Linux user, you can use the `sudo` command to gain root privileges and then run the following command to add the above line to your `/etc/hosts` file:

```bash
sudo echo "172.18.1.100  <domain>" | sudo tee -a /etc/hosts
```

See the [domain-registry documentation](https://www.nebari.dev/docs/how-tos/domain-registry#what-is-a-dns) for details.

:::

### Exposing container network (for MacOS)

Docker for macOS does not expose container networks directly on the MacOS host, for this we will use
[docker-mac-net-connect](https://github.com/chipmk/docker-mac-net-connect), which lets you connect directly
to Docker-for-Mac containers via IP address. You can install and start it with following command:

```bash
# Install via Homebrew
$ brew install chipmk/tap/docker-mac-net-connect

# Run the service and register it to launch at boot
$ sudo brew services start chipmk/tap/docker-mac-net-connect
```

### Docker Images

You can skip this section if you have an x86_64 machine. If you're using Mac M1, then the x86_64 docker images
will not work out of the box. You would need to use images that are built with support for arm as well.

We're building all the docker images for both platforms except external images. The only external image relevant
here is keycloak. You'd need to update the keycloak image for the deployment, which can be done by adding the
override for the keycloak deployment to update the image:

```yaml
security:
  keycloak:
    initial_root_password: <SANITIZED>
    overrides:
      image:
        repository: quay.io/nebari/keycloak
        tag: sha-b4a2d1e
```

### Increase fs watches

Depending on your host system, you may need to increase the `fs.inotify.max_user_watches` and
`fs.inotify.max_user_instances kernel parameters` if you see the error "too many open files" in the logs of
a failing pod.

```bash
sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl fs.inotify.max_user_instances=512
```

See the [kind troubleshooting
docs](https://kind.sigs.k8s.io/docs/user/known-issues/#pod-errors-due-to-too-many-open-files) for more information.

## Deploying Nebari

With the `nebari-config.yaml` configuration file now created, Nebari can be deployed for the first time with:

```bash
nebari deploy -c nebari-config.yaml --disable-prompt
```

If the deployment is successful, you will see the following output:

```bash
[terraform]: Nebari deployed successfully
Services:
 - argo-workflows -> https://domain/argo/
 - conda_store -> https://domain/conda-store/
 - dask_gateway -> https://domain/gateway/
 - jupyterhub -> https://domain/
 - keycloak -> https://domain/auth/
 - monitoring -> https://domain/monitoring/
Kubernetes kubeconfig located at file:///tmp/NEBARI_KUBECONFIG
Kubecloak master realm username=root *****
...
```

### Verify the local deployment

Finally, if everything is set properly you should be able to cURL the JupyterHub Server. Run

```
curl -k https://domain/hub/login
```

It's also possible to visit `https://domain` in your web browser to select the deployment.
As default for a local deployment the https certificates generated during deployments aren't signed by a recognized [Certificate Authority (CA)](https://en.wikipedia.org/wiki/Certificate_authority) and are self-signed by [Traefik](https://github.com/traefik/traefik) instead.

Several browsers makes it difficult to view a self-signed certificate that are not added to the certificate registry. So, if you do not want to use Let's Encrypt, you can use the following workarounds to properly view the pages:

A workaround for Firefox:

- Visit **about:config** and change the `network.stricttransportsecurity.preloadlist` to `false`

And a workaround for Chrome:

- Type **badidea** or **thisisunsafe** while viewing the rendered page (this has to do with how [Chrome preloads some domains for its HTTP Strict Transport Security list](https://hstspreload.org/) in a way that can't be manually removed)

### Using Let's Encrypt Certificates

If your "local" deployment happens to be exposed to the internet (e.g. with a load balancer deployed and managed outside of Nebari) and you are able to set up a valid public DNS record, you can instead use Let's Encrypt to provision trusted TLS certificates. For more in-depth on DNS records and how Nebari handles their configuration, you can visit our [Domain Registry](/docs/how-tos/domain-registry) documentation.

To switch the default behavior and use a [Let's Encrypt](https://letsencrypt.org/) signed certificate instead, you can update the following section in your `nebari-config.yaml` file, and then re-run `nebari deploy` as shown above:

```yaml
certificate:
  type: lets-encrypt
  acme_email: <your-email-address>
  acme_server: https://acme-v02.api.letsencrypt.org/directory
```

Note the above snippet can be automatically provisioned in your configuration if you provided the `--ssl-cert-email` flag when you ran `nebari init`.

Let's Encrypt heavily rate limits their production endpoint. In order to avoid throttling, Nebari's traefik deployments will [store certificates in an acme.json file](https://doc.traefik.io/traefik/https/acme/#storage) for the duration of their validity. Nebari will mount a PVC and save the file on the container at the location `/mnt/acme-certificates/acme.json`.

:::note
In order to refresh the certificate before it is invalidated, you will need to delete the `acme.json` file then restart the Traefik deployment by deleting the existing pod and letting a new one spin up. This may be necessary if you change the domain name of your Nebari deployment.
:::

## Destroying Nebari

Nebari also has a `destroy` command that works the same way the deploy works but instead of creating the provisioned resources it destroys it.

```bash
nebari destroy -c nebari-config.yaml
```

To see all the options available for the destroy command, type the following command on your command line:

```bash
nebari destroy --help
```

![A representation of the output generated when nebari deploy help command is executed.](/img/how-tos/nebari-destroy-help.png)



---
File: /nebari-docs/docs/docs/how-tos/nebari-stages-directory.md
---

---
id: nebari-stages-directory
title: Nebari Stages Directory
description: Purpose and Usage of the "stages" Directory
---

# The Nebari "Stages" Directory

The Nebari CLI generates Terraform manifests and saves them on disk in directories `stages/[stage-name]` relative to the location of your configuration file. Knowing how and when these directories are used will allow you to better track changes that the Nebari CLI makes to your running Nebari deployment when you upgrade or modify the configuration file.

## Nebari CLI Steps

The following three Nebari CLI commands trigger different steps in the Nebari deployment process.

`nebari validate` will check whether your configuration file is valid for the Nebari schema of the Nebari CLI version that you are using. It will **NOT** render new `stages` directories.

`nebari render` will validate your configuration file **AND** generate new Terraform manifests in the `stages` directories but will **NOT** deploy those changes.

`nebari deploy` will validate your configuration file, generate new Terraform manifests **UNLESS** you use the `--disable-render`, **AND** deploy those changes. If `--disable-render` is set, the Nebari CLI will instead use whatever code is present in the existing `stages` directory. This is useful to ensure that only Terraform code changes that you have seen are present with the deploy (see Tip below).

:::tip
You can also use `nebari info` to generate a view a list of 'Runtime Stage Ordering' which will include all of the default stages included in your Nebari version along with any additionally installed extensions.
:::

:::tip
If you are using a `git` repo to maintain your Nebari configuration, you can preview the effects that a Nebari version upgrades and/or config file change will make to your running Nebari deployment. First perform the [Nebari upgrade command][upgrade-nebari] or modify your `nebari-config.yaml` file, then run `nebari render`. After the new `/stages/` directories are generated, use `git diff` to see any Terraform changes that will be made with the next `nebari deploy`.

Please note that some changes (e.g. container image tags) are injected as Terraform variable values at runtime by `nebari deploy`, this technique will not capture every possible change the way a basic `terraform plan` would.
:::

## Overview of Nebari Stages

When you run `nebari deploy`, you will see several log entries indicating Terraform initializing anew. This is because **_every Nebari stage is an independent Terraform deployment._** This modular architecture allows for the greatest flexibility when customizing your platform, such as using different cloud providers in the Infrastructure stage, overriding or skipping entire stages, or adding [extensions][nebari-extension-system] to your deployment. As the `deploy` runs, the deployer will track the Terraform outputs of each stage's Terraform deployment and append them to a `stage_outputs` object so that subsequent stages can access the data as Terraform inputs.

- **01-terraform-state/[provider-name]** - Creates the infrastructure for Terraform backend, such as Google Cloud Storage bucket or AWS S3 bucket. If you are using a `local` or `existing` provider, this stage is skipped and Terraform instead keeps a local `.tfstate` file in each stage folder.
- **02-infrastructure/[provider-name]** - Deploys the underlying infrastructure, notably the networking components and Kubernetes cluster (unless using the `existing` provider). This is the last core Nebari stage which directly interacts with the Terraform cloud provider.
- **03-kubernetes-initialize** - Sets up fundamental Kubernetes objects (namespaces, autoscalers, etc.)
- **04-kubernetes-ingress** - Deploys the Traefik ingress. Note that after this stage, your Nebari CLI deployer will attempt to connect with your cluster. After this stage your Nebari CLI deployer will attempt to communicate with your cluster, so [DNS must be provisioned][setup-dns].
- **05-kubernetes-keycloak** - Deploys Keyclaok for authentication.
- **06-kubernetes-keycloak-configuration** - Applies Nebari-specific configurations to Keycloak.
- **07-kubernetes-services** - Deploys all of the core Nebari components that rely on Keycloak, notably JupyterHub and Conda Store. _The vast majority of the end-user Nebari functionality is deployed in this stage._
- **08-nebari-extensions** - This is an experimental stage that can be used to deploy custom Helm charts or define new Kubernetes services to inject into Nebari directly from your configuration file. This is **_NOT_** related to the [Nebari Extension System][nebari-extension-system].

If you have installed any [Nebari Python extensions][nebari-extension-system], they will also be rendered as folders in the stages directory.

<!-- internal links -->

[setup-dns]: /how-tos/domain-registry.md
[upgrade-nebari]: /how-tos/nebari-upgrade.md
[nebari-extension-system]: /how-tos/nebari-extension-system.md



---
File: /nebari-docs/docs/docs/how-tos/nebari-upgrade.md
---

---
id: nebari-upgrade
title: Upgrade Nebari
description: An overview of how to upgrade Nebari. Includes instructions on updating when breaking changes are expected to occur.
---

# Upgrading Nebari

:::note
This is a guide to upgrade Nebari to a newer version. If you are migrating from QHub (deprecated) to Nebari, you can find instructions [here](#migrating-from-qhub-deprecated-to-nebari). If you are updating from an older version of QHub to another version of QHub, those instructions are [here](#upgrade-qhub-from-pre-040-to-another-qhub-version).
:::

## Upgrade Nebari

### Backup existing data

Perform manual backups of the NFS data and JupyterHub database.

It is strongly recommended you [backup your data](./manual-backup.md) before upgrading!

### Install latest version of Nebari

Use pip or conda to install the latest Nebari version.

```shell
pip install --upgrade nebari
```

or

```shell
conda update nebari
```

### Nebari upgrade command

Run the following to upgrade Nebari.

```shell
nebari upgrade -c nebari-config.yaml
```

The upgrade command may print some manual instructions that need to be run before continuing with the upgrade. If so, complete those instructions before continuing on.

:::info
New versions of Nebari often include support for newer versions of Kubernetes, as such you might need to upgrade your cluster's Kubernetes version. Please refer to [Upgrade Kubernetes version docs](./upgrade-kubernetes-version.md) for more details.
:::

### Re-deploy Nebari

If you are deploying Nebari from your local machine (see [this section](#cicd-render-and-commit-to-git) for CI/CD deployments), you will now have a `nebari-config.yaml` file that you can deploy.

```shell
nebari deploy -c nebari-config.yaml
```

### CI/CD: render and commit to git

For CI/CD (GitHub/GitLab) workflows, you will need to regenerate the workflow files based on the latest `Nebari` version's templates.

With the newly upgraded `nebari-config.yaml` file, run:

```shell
nebari render -c nebari-config.yaml
```

(Note that `nebari deploy` would perform this render step too, but will also immediately redeploy your Nebari.)

Commit all the files (`nebari-config.yaml` and GitHub/GitLab workflow files) back to the remote repo. All files need to be committed together in the same commit.

## Migrating from QHub (deprecated) to Nebari

:::note
If you are migrating your package from QHub to Nebari there are a couple steps in the upgrade process to make note of. First, you will re-name anything labeled with `qhub` with `nebari`. Second, the version system used for Nebari is different from QHub. Nebari uses a CalVer version system, starting at version 2022.10.1. QHub was tracked with a SemVer version system, the last version being 0.4.5.
:::

### Backup existing data

Always [backup your data](./manual-backup.md) before upgrading!

### Locate configuration file

Locate your `qhub-config.yaml` configuration file.

For CI/CD deployments, you will need to `git clone <repo URL>` into a folder on your local machine if you haven't done so already.

:::warning Breaking upgrades

### Rename existing QHub URL

This will allow the existing cluster to remain alive in case it is needed, but the idea would be not to have it in use from now on.

Move the `qhub-config.yaml` to `nebari-config.yaml` and change the following lines:

```yaml
project_name: myqhub
domain: qhub.myproj.com
```

could become:

```yaml
project_name: mynebari
domain: nebari.myproj.com
```

Run the command `nebari deploy` using the existing (old) version of QHub.
:::

### Upgrade the `QHub` command package

To install (or upgrade) to a specific version of the python package used to manage Nabari, you can run the following, choosing either pip or conda:

```shell
pip install --upgrade nebari
```

or

````shell
conda update nebari

### Upgrade `nabari-config.yaml` file

In the folder containing your qhub configuration file, run:

```shell
nebari upgrade -c nebari-config.yaml
````

:::warning Breaking upgrades
For those that are upgrading from an older version (e.g. v0.3.14 (or earlier)), you will need to go back to the original `qhub-config.yaml` file which contained the original domain.
:::

Upgrading will output a newer version of `nebari-config.yaml` that is compatible with the updated version of `Nebari`. The process outputs a list of changes it has made. The `upgrade` command creates a copy of the original unmodified config file (`qhub-config.yaml.old.backup`) as well as any other files that may be required by the upgraded cluster (including a JSON file (`nabari-users-import.json`) used to import existing users into Keycloak if they are not already there).

### Validate special customizations in `nebari-config.yaml`

You may have made special customizations to your `nebari-config.yaml`, such as using your own versions of Docker images. Please check your `nebari-config.yaml` and decide if you need to update any values that would not have been changed automatically - or, for example, you may need to build new versions of your custom Docker images to match any changes in Nebari's images.

:::warning Breaking upgrades

### Rename the Project and Increase Kubernetes version

You need to rename the project to avoid clashes with the existing (old) cluster which would otherwise already own resources based on the names that the new cluster will attempt to use.

The domain should remain as the preferred main one that was always in use previously.

For example:

```yaml
project_name: myqhub
domain: qhub.myproj.com
```

could become:

```yaml
project_name: mynebari
domain: nebari.myproj.com
```

It is also a good time to upgrade your version of Kubernetes. Look for the `kubernetes_version` field within the cloud provider section of the `nebari-config.yaml` file and increase it to the latest.
:::

### Redeploy QHub

For local deployments, run the following:

```shell
nebari deploy -c qhub-config.yaml
```

### CI/CD: render and commit to git

For CI/CD (GitHub/GitLab) workflows, update the workflow files.
Run the following with the updated `nebari-config.yaml` file:

```shell
nebari render -c nebari-config.yaml
```

Commit all the files (`nebari-config.yaml` and GitHub/GitLab workflow files) back to the remote repo. All files need to be committed together in the same commit.

:::warning Breaking upgrades
For upgrades from older versions to v0.4, you will need to do the following steps. Everyone else is done!

### Update OAuth callback URL for Auth0 or GitHub

If your QHub deployment relies on Auth0 or GitHub for authentication, please update the OAuth callback URL.

<details>
<summary>Expand this section for Auth0 instructions </summary>

1. Navigate to the your Auth0 tenancy homepage and from there select "Applications".

2. Select the "Regular Web Application" with the name of your deployment.

3. Under the "Application URIs" section, paste the new OAuth callback URL in the "Allowed Callback URLs" text block. The URL should be `https://{your-nebari-domain}/auth/realms/nebari/broker/auth0/endpoint`, replacing `{your-nebari-domain}`with your literal domain of course.

</details>

<details>
<summary>Expand this section for GitHub auth instructions </summary>

1. Go to the [GitHub Developer Settings](https://github.com/settings/developers).

2. Click "OAuth Apps" and then click the app representing your Nebari instance.

3. Under "Authorization callback URL", paste the new GitHub callback URL. The URL should be
   `https://{your-nebari-domain}/auth/realms/nebari/broker/github/endpoint`, replacing `{your-nebari-domain}` with your literal domain of course.

</details>

### Restore from Backups

Next, you will need to perform the following steps to restore from a previously generated backup, as described in the [Manual Backups documentation](./manual-backup.md):

1. Restore the NFS data from your S3 (or similar) backup
2. Immediately after restoring NFS data, you must run some extra commands as explained in the backup/restore docs for v0.4 upgrades specifically.
3. Restore the JupyterHub SQLite database.

### Import users into Keycloak

The last two steps are:

1. Change the Keycloak `root` user password, documented [here](./configure-keycloak-howto.md#change-keycloak-root-password)
2. Import existing users, documented [here](./manual-backup.md#import-keycloak).

For more details on this process, visit the [Keycloak docs section][login-keycloak].

### Known versions that require re-deployment

Version `v0.3.11` on AWS has an error with the Kubernetes config map. See [this GitHub discussion related to AWS K8s config maps](https://github.com/Quansight/nebari/discussions/841) for more details.
:::

## Upgrade QHub from pre-0.4.0 to another QHub version

:::warning Breaking upgrades
If you are upgrading from an older version, (e.g. v0.3.14 (or earlier), to v0.4), the cluster cannot be upgraded in-situ so you must perform a redeployment. Please pay extra attention to the highlighted `Breaking upgrades` steps throughout the process!
:::

### Backup existing data

Perform manual backups of the NFS data and JupyterHub database.

:::warning Breaking upgrades
For older versions of QHub, ignore the section about Keycloak data since that will not exist in older (v0.3.14) clusters.
:::

Always [backup your data](./manual-backup.md) before upgrading!

### Locate configuration file

Locate your `qhub-config.yaml` configuration file.

For CI/CD deployments, you will need to `git clone <repo URL>` into a folder on your local machine if you haven't done so already.

:::warning Breaking upgrades

### Rename existing QHub URL

This will allow the existing cluster to remain alive in case it is needed, but the idea would be not to have it in use from now on.

In the `qhub-config.yaml` for example:

```yaml
project_name: myqhub
domain: qhub.myproj.com
```

could become:

```yaml
project_name: myqhub
domain: qhub-old.myproj.com
```

Run the command `nebari deploy` using the existing (old) version of QHub.
:::

### Upgrade the `QHub` command package

To install (or upgrade) to a specific version of the python package used to manage QHub, you can run the following, choosing either pip or conda:

```shell
pip install --upgrade qhub==<version>
```

or

```shell
conda install qhub=<version>
```

### Upgrade `qhub-config.yaml` file

In the folder containing your qhub configuration file, run:

###TO DO: is there a QHub command?

```shell
nebari upgrade -c nebari-config.yaml
```

:::warning Breaking upgrades
For those that are upgrading from an older version (e.g. v0.3.14 (or earlier)), you will need to go back to the original `qhub-config.yaml` file which contained the original domain.
:::

Upgrading will output a newer version of `qhub-config.yaml` that is compatible with the updated version of `QHub`. The process outputs a list of changes it has made. The `upgrade` command creates a copy of the original unmodified config file (`qhub-config.yaml.old.backup`) as well as any other files that may be required by the upgraded cluster (including a JSON file (`qhub-users-import.json`) used to import existing users into Keycloak if they are not already there).

### Validate special customizations to `qhub-config.yaml`

You may have made special customizations to your `qhub-config.yaml`, such as using your own versions of Docker images. Please check your `qhub-config.yaml` and decide if you need to update any values that would not have been changed automatically - or, for example, you may need to build new versions of your custom Docker images to match any changes in QHub's images.

:::warning Breaking upgrades

### Rename the Project and Increase Kubernetes version

You need to rename the project to avoid clashes with the existing (old) cluster which would otherwise already own resources based on the names that the new cluster will attempt to use.

The domain should remain as the preferred main one that was always in use previously.

For example:

```yaml
project_name: myqhub
domain: qhub.myproj.com
```

could become:

```yaml
project_name: myqhubnew
domain: qhub.myproj.com
```

It is also a good time to upgrade your version of Kubernetes. Look for the `kubernetes_version` field within the cloud provider section of the `qhub-config.yaml` file and increase it to the latest.
:::

### Redeploy QHub

For local deployments, run the following:
###TO DO: is there a QHub command?

```shell
nebari deploy -c qhub-config.yaml
```

At this point you may see an error message saying that deployment is prevented due to the `prevent_deploy` setting in your YAML file. This is a safeguard to ensure that you only proceed if you are aware of possible breaking changes in the current upgrade.

For example, we may be aware that you will lose data due to this upgrade, so need to note a specific upgrade process to keep your data safe. Always check the release notes of the release in this case and get in touch with us if you need assistance. For example, you may find that your existing cluster is intentionally deleted so that a new replacement can be deployed instead, in which case your data must be backed up so it can be restored after the upgrade.

**When you are ready, remove the `prevent_deploy` setting from the config file, and run the `nebari deploy -c nebari-config.yaml` again.**

### CI/CD: render and commit to git

For CI/CD (GitHub/GitLab) workflows, update the workflow files.
Run the following with the updated `nebari-config.yaml` file:

###TO DO: is there a QHub command?

```shell
nebari render -c nebari-config.yaml
```

Commit all the files (`qhub-config.yaml` and GitHub/GitLab workflow files) back to the remote repo. All files need to be committed together in the same commit.

:::warning Breaking upgrades
For upgrades from older versions to v0.4, you will need to do the following steps. Everyone else is done!

### Update OAuth callback URL for Auth0 or GitHub

If your QHub deployment relies on Auth0 or GitHub for authentication, please update the OAuth callback URL.

<details>
<summary>Expand this section for Auth0 instructions </summary>

1. Navigate to the your Auth0 tenancy homepage and from there select "Applications".

2. Select the "Regular Web Application" with the name of your deployment.

3. Under the "Application URIs" section, paste the new OAuth callback URL in the "Allowed Callback URLs" text block. The URL should be `https://{your-nebari-domain}/auth/realms/nebari/broker/auth0/endpoint`, replacing `{your-nebari-domain}`with your literal domain of course.

</details>

<details>
<summary>Expand this section for GitHub auth instructions </summary>

1. Go to the [GitHub Developer Settings](https://github.com/settings/developers).

2. Click "OAuth Apps" and then click the app representing your Nebari instance.

3. Under "Authorization callback URL", paste the new GitHub callback URL. The URL should be
   `https://{your-nebari-domain}/auth/realms/nebari/broker/github/endpoint`, replacing `{your-nebari-domain}` with your literal domain of course.

</details>

### Restore from Backups

Next, you will need to perform the following steps to restore from a previously generated backup, as described in the [Manual Backups documentation](./manual-backup.md):

1. Restore the NFS data from your S3 (or similar) backup
2. Immediately after restoring NFS data, you must run some extra commands as explained in the backup/restore docs for v0.4 upgrades specifically.
3. Restore the JupyterHub SQLite database.

### Import users into Keycloak

The last two steps are:

1. Change the Keycloak `root` user password, documented [here](./configure-keycloak-howto.md#change-keycloak-root-password)
2. Import existing users, documented [here](./manual-backup.md#import-keycloak).

For more details on this process, visit the [Keycloak docs section][login-keycloak].

### Known versions that require re-deployment

Version `v0.3.11` on AWS has an error with the Kubernetes config map. See [this GitHub discussion related to AWS K8s config maps](https://github.com/Quansight/nebari/discussions/841) for more details.
:::

<!-- Internal links -->

[login-keycloak]: /docs/tutorials/login-keycloak



---
File: /nebari-docs/docs/docs/how-tos/setup-argo.md
---

---
id: setup-argo
title: Set up Argo workflows
description: Argo Workflow Setup
---

# Set up Argo workflows

Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo
workflows comes enabled by default with Nebari deployments.

## Access Argo Server

If Argo Workflows is enabled, users can access argo workflows server at: `your-nebari-domain.com/argo`. Log in via
Keycloak with your usual credentials.

## Overrides of Argo Workflows Helm Chart values

Argo Workflows is deployed using the Argo Workflows Helm Chart. The values.yaml for the helm chart can be overridden as
needed via the overrides flag. The default values file can be found
[here](https://github.com/argoproj/argo-helm/blob/argo-workflows-0.22.9/charts/argo-workflows/values.yaml). For example,
the following could be done to add additional environment variables to the controller container.

```yaml
argo_workflows:
  enabled: true
  overrides:
    controller:
      extraEnv:
        - name: foo
          value: bar
```

## Nebari Workflow Controller (Beta)

Nebari includes an admission controller for Argo Workflows that 1) prevents users from mounting shared directories they don't have permissions for in their Workflows and 2) provides a convenient way to start an Argo Workflow with conda envs and shared directories mounted that the user does have permissions for. Valid workflows should not be affected, however submitting Workflows via `kubectl apply` on a kubernetes manifest is not supported when Nebari Workflow Controller is enabled. Submitting Argo Workflows through other methods (Argo CLI, Argo UI, Hera, etc.) will work as expected. If you'd like to disable Nebari Workflow Controller for any reason, you can do so in the nebari-config.yaml file.

```yaml
argo_workflows:
  enabled: true
  nebari_workflow_controller:
    enabled: false
```

## Disable Argo Workflows

To turn off the cluster monitoring on Nebari deployments, simply turn off the feature flag within your
`nebari-config.yaml` file. For example:

```yaml
argo_workflows:
  enabled: false
```

Refer to the [Argo documentation](https://argoproj.github.io/argo-workflows/) for further details on Argo Workflows.



---
File: /nebari-docs/docs/docs/how-tos/setup-healthcheck.md
---

---
id: setup-healthcheck
title: Set up healthchecks with Kuberhealthy
description: Set up healthchecks with Kuberhealthy
---

:::warning

This feature is in beta status. It should be used with caution.

:::

# Overview

Nebari integrates [Kuberhealthy](https://kuberhealthy.github.io/kuberhealthy/) to perform internal healthchecks on Nebari. This is an extensible Kubernetes native framework for continuous synthetic testing. Kuberhealthy is set up to export metrics to Prometheus. This allows them to be seen in Grafana.

## Enabling

Healthchecks are currently considered a beta feature that we are testing. Due to this, they are disabled by default. To enable healthchecks, add the following configuration under the `monitoring` configuration in your `nebari-config.yaml`.

```yaml
monitoring:
  healthchecks:
    enabled: true
```

## Checking status of Healthchecks

All healthchecks are exported as metrics to Prometheus and can be viewed in Grafana.

For example: To see the uptime for the conda-store service, you can run:

```
1 - (sum(count_over_time(kuberhealthy_check{check="dev/conda-store-http-check", status="0"}[30d])) OR vector(0))/(sum(count_over_time(kuberhealthy_check{check="dev/conda-store-http-check", status="1"}[30d])) * 100)
```

in Grafana, which will show you the following chart.

![Grafana chart showing the uptime for conda store](/img/how-tos/nebari-healthchecks.png)

To see what other healthchecks are available, you can use the metric explorer in Grafana. Select the metric type of `kuberhealthy_check` and the label filter of `check`. The values list will be a list of the checks that
have metrics available.

![Display of available kuberhealthy metrics in Grafana](/img/how-tos/nebari-healthchecks1.png)

:::note

If you have previously deployed Nebari without healthchecks, You may need to restart your Prometheus service to get it to pick up the kuberhealthy metrics.

:::

## Summary of available healthchecks

Below is an explanation of the available healthchecks. This list may not be comprehensive as work on this feature is ongoing.

| <div style={{width:180}}>Check Label</div> | Description                                                                                                                                                     |
| ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| conda-store-http-check                     | verifies that conda-store is accessible via it's REST API                                                                                                       |
| jupyterhub-http-check                      | verifies JupyterHub is running                                                                                                                                  |
| dns-status-internal                        | verifies internal DNS is accessible                                                                                                                             |
| daemonset                                  | verifies that a daemonset can be created, fully provisioned, and torn down. This checks the full kubelet functionality of every node in your Kubernetes cluster |
| deployment                                 | verifies that a fresh deployment can run, deploy multiple pods, pass traffic, do a rolling update (without dropping connections), and clean up successfully     |
| keycloak-http-check                        | verifies Keycloak is accessible                                                                                                                                 |



---
File: /nebari-docs/docs/docs/how-tos/setup-monitoring.md
---

# How to Set Up Monitoring on Nebari

In Nebari, we've integrated Grafana, Prometheus, and Loki to provide robust monitoring capabilities for
your data science platform. This integration allows you to visualize metrics, monitor system health, and
analyze logs effectively. Below, we'll discuss each component and how they are deployed using Helm charts,
along with instructions on how to override configuration values.

Monitoring is enabled by default in Nebari. It can be disabled by setting the following in your `nebari-config.yaml`.

```yaml
monitoring:
  enabled: false
```

## Components Overview

### Grafana

[Grafana](https://grafana.com/) is a leading open-source platform for monitoring and observability.
It provides rich visualization tools and dashboards for analyzing and monitoring metrics from various data sources.

### Prometheus

[Prometheus](https://prometheus.io/) is a popular open-source monitoring and alerting toolkit. It collects
metrics from configured targets, stores them efficiently, and allows querying them in real-time.

### Loki

[Loki](https://grafana.com/docs/loki/latest/) is a horizontally-scalable, highly available, multi-tenant log
aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate, as it
does not index the contents of the logs, but rather a set of labels for each log stream.

See [How to access system logs (Loki) via Grafana][access-logs-loki] for more information on using Loki in Nebari.

## Terraform Overrides

Nebari provides its users with the ability to customize the deployment of various component
and Loki is one of them. Loki deployment is made up of three fundamental components:

- Loki: a set of components that when composed forms a fully featured logging stack
- Promtail: an agent which ships the contents of local logs to a Loki instance
- MinIO: a Kubernetes-native high-performance object storage server which is designed for large-scale
  private cloud infrastructure and compatible with Amazon S3.

```yaml
monitoring:
  enabled: true
  overrides:
    loki: <LOKI-HELM-CHART-VALUES-OVERRIDE>
    promtail: <PROMTAIL-HELM-CHART-VALUES-OVERRIDE>
    minio: <MINIO-HELM-CHART-VALUES-OVERRIDE>
```

Below are some examples of customizing your loki deployment via terraform overrides:

### Log Storage Config

We use minio for logs storage as default, it is a high-performance, S3 compatible object store. You can use
any S3 compatible object store instead of minio, below is an example configuration for using AWS S3:

```yaml
monitoring:
  enabled: true
  minio_enabled: false
  overrides:
    loki:
      loki:
        storage_config:
          aws:
            s3: s3://<access_key>:<uri-encoded-secret-access-key>@<region>
```

See https://grafana.com/docs/loki/latest/storage/#aws-deployment-s3-single-store for more configuration
options.

### Log Storage

You can customize the size of persistent volume for logs storage, by setting up the value for
`minio.persistence.size` as shown below:

```yaml
monitoring:
  enabled: true
  overrides:
    minio:
      persistence:
        size: 100Gi
```

### Log Retention Period

Since the storage for MinIO is predefined and is not infinitely scalable out-of-the-box like
cloud storage solutions like S3, GCS, etc, unless you keep increasing minio persistent volume size.
It is a good idea to clear up old logs to make space for new ones, Nebari does this by default with
60 days retention period. You can customize the log retention period by overriding the default
value as follows:

```yaml
monitoring:
  enabled: true
  overrides:
    loki:
      loki:
        limits_config:
          # The minimum retention period is 24h.
          retention_period: 90d
```

This may not be a suitable solution for many cases, like if your organization needs to keep
all the logs forever, then you may want to use a very large persistent volume and disable
deletion of old logs. You can disable deletion via:

```yaml
monitoring:
  enabled: true
  overrides:
    loki:
      loki:
        compactor:
          retention_enabled: false
```

## Logging architecture

The architecture diagram below shows a simplified, high level explanation of the logging components on Nebari.

![Grafana](/img/how-tos/grafana-loki-promtail-architecture.png)

`Grafana` is the dashboarding user interface which allows us to use `Loki` as the data source for our logs. `Loki` connects to [`promtail`](https://grafana.com/docs/loki/latest/send-data/promtail/) as it's source.

The `promtail` component scrapes logs from various pods on the kubernetes nodes. The `kube api server` provides the API endpoints which `promtail` uses for for discovering and scraping its targeted resources

End users viewing the logs in `Grafana` will create queries using `Loki` as the data source, typically querying based on `labels`. However, it's important to note that Grafana labels differ from Kubernetes labels, as their main goal is to act as an aggregation layer of logs from multiple matching resources into a single "stream," allowing users to easily access a collection of logs from various Kubernetes resources with just a single logical label.

:::note
Loki's "labels" are used to filter collections of logs from the available [kubernetes_sd](https://grafana.com/docs/loki/latest/send-data/promtail/configuration/#kubernetes_sd_config) API endpoints, in a similar way as to how Prometheus handles metrics. These labels are configured through Promtail, which is the agent responsible for collecting and shipping logs to Loki, based on the defined [targets](https://grafana.com/docs/loki/latest/send-data/promtail/configuration/#scrape_configs) and scraping configurations.
:::

For details on how to view specific logs in Loki, check out the document ["How to access system logs (Loki) via Grafana"](access-logs-loki)

## References

[More information on promtail configurations](https://grafana.com/docs/loki/latest/send-data/promtail/configuration/)
[Understanding labels in Loki](https://grafana.com/docs/loki/latest/get-started/labels/#understand-labels)

<!-- Internal links -->

[access-logs-loki]: /how-tos/access-logs-loki.md



---
File: /nebari-docs/docs/docs/how-tos/telemetry.md
---

# Telemetry

:::note
Nebari does NOT collects telemetry data from its installations.
This document is for using telemetry for users/organization's
private usage/visualization/processing of data for JupyterLab events.
:::

Telemetry in Nebari enables the collection, processing, and visualization of data
related to Jupyter Lab events. This documentation provides a guide on setting up
and utilizing telemetry features effectively.

Nebari uses [jupyterlab-pioneer](https://jupyterlab-pioneer.readthedocs.io/en/latest/) for
enabling telemetry on JupyterLab usage data.

## JupyterLab Pioneer

JupyterLab Pioneer is a JupyterLab extension for generating and exporting JupyterLab event
telemetry data.

## Installation

Telemetry can be enabled on Nebari via JupyterLab Pioneer by adding the following in the
`nebari-config.yml`:

```yaml
telemetry:
  jupyterlab_pioneer:
    enabled: true
```

## Configuration

Since the telemetry events for JupyterLab is emitted in the pod logs, you may want to configure
the format for the same. This can configure logging format via `log_format` variable:
The syntax for this is same as the `format` param in python's `logging.basicConfig` method,
which is defined in the docs [here](https://docs.python.org/3/howto/logging.html#changing-the-format-of-displayed-messages).

```yaml
telemetry:
  jupyterlab_pioneer:
    enabled: true
    log_format: "%(asctime)s %(levelname)9s %(lineno)4s %(module)s: %(message)s"
```

## Usage

Enabling `jupyterlab_pioneer` in `telemetry` will start emitting telemetry data in the JupyterLab pods,
which can be seen either via [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) or
[`k9s`](https://k9scli.io/). Below are some logs for some jupyterlab events.

```
2023-11-14 12:00:29.039 {"eventDetail": {"eventName": "ActiveCellChangeEvent", "eventTime": 1699962682425, "eventInfo": {"cells": [{"id": "83a72108-db56-43e3-b938-4aca6667484f", "index": 5}]}}, "notebookState": {"sessionID": "09ceefc2-82af-41d0-b806-a76369748020", "notebookPath": "alpha.ipynb", "notebookContent": null}}
2023-11-14 12:00:29.039 {"eventDetail": {"eventName": "CellEditEvent", "eventTime": 1699962682296, "eventInfo": {"index": 6, "doc": [""]}}, "notebookState": {"sessionID": "09ceefc2-82af-41d0-b806-a76369748020", "notebookPath": "alpha.ipynb", "notebookContent": null}}
2023-11-14 12:00:29.039 {"eventDetail": {"eventName": "CellRemoveEvent", "eventTime": 1699962682286, "eventInfo": {"cells": [{"index": 7}]}}, "notebookState": {"sessionID": "09ceefc2-82af-41d0-b806-a76369748020", "notebookPath": "alpha.ipynb", "notebookContent": null}}
2023-11-14 12:00:29.039 {"eventDetail": {"eventName": "ActiveCellChangeEvent", "eventTime": 1699962682284, "eventInfo": {"cells": [{"id": "2f4e6013-6729-4e01-8419-4feeda64cd48", "index": 6}]}}, "notebookState": {"sessionID": "09ceefc2-82af-41d0-b806-a76369748020", "notebookPath": "alpha.ipynb", "notebookContent": null}}
2023-11-14 12:00:29.039 {"eventDetail": {"eventName": "CellEditEvent", "eventTime": 1699962682122, "eventInfo": {"index": 7, "doc": [""]}}, "notebookState": {"sessionID": "09ceefc2-82af-41d0-b806-a76369748020", "notebookPath": "alpha.ipynb", "notebookContent": null}}
2023-11-14 12:00:29.039 {"eventDetail": {"eventName": "CellRemoveEvent", "eventTime": 1699962682112, "eventInfo": {"cells": [{"index": 8}]}}, "notebookState": {"sessionID": "09ceefc2-82af-41d0-b806-a76369748020", "notebookPath": "alpha.ipynb", "notebookContent": null}}
2023-11-14 12:00:29.039 {"eventDetail": {"eventName": "ActiveCellChangeEvent", "eventTime": 1699962682109, "eventInfo": {"cells": [{"id": "8edeabdf-fdd1-4465-af5b-ca13c314eea4", "index": 7}]}}, "notebookState": {"sessionID": "09ceefc2-82af-41d0-b806-a76369748020", "notebookPath": "alpha.ipynb", "notebookContent": null}}
2023-11-14 12:00:29.039 {"eventDetail": {"eventName": "CellEditEvent", "eventTime": 1699962681958, "eventInfo": {"index": 8, "doc": [""]}}, "notebookState": {"sessionID": "09ceefc2-82af-41d0-b806-a76369748020", "notebookPath": "alpha.ipynb", "notebookContent": null}}
2023-11-14 12:00:29.039 {"eventDetail": {"eventName": "CellRemoveEvent", "eventTime": 1699962681950, "eventInfo": {"cells": [{"index": 9}]}}, "notebookState": {"sessionID": "09ceefc2-82af-41d0-b806-a76369748020", "notebookPath": "alpha.ipynb", "notebookContent": null}}
2023-11-14 12:00:29.039 {"eventDetail": {"eventName": "ActiveCellChangeEvent", "eventTime": 1699962681948, "eventInfo": {"cells": [{"id": "6b8d84a5-c72c-4c8e-9125-39bd8cf0bee5", "index": 8}]}}, "notebookState": {"sessionID": "09ceefc2-82af-41d0-b806-a76369748020", "notebookPath": "alpha.ipynb", "notebookContent": null}}
2023-11-14 12:00:29.039 {"eventDetail": {"eventName": "CellEditEvent", "eventTime": 1699962681786, "eventInfo": {"index": 9, "doc": [""]}}, "notebookState": {"sessionID": "09ceefc2-82af-41d0-b806-a76369748020", "notebookPath": "alpha.ipynb", "notebookContent": null}}
2023-11-14 12:00:29.039 {"eventDetail": {"eventName": "CellRemoveEvent", "eventTime": 1699962681775, "eventInfo": {"cells": [{"index": 10}]}}, "notebookState": {"sessionID": "09ceefc2-82af-41d0-b806-a76369748020", "notebookPath": "alpha.ipynb", "notebookContent": null}}
2023-11-14 12:00:29.039 {"eventDetail": {"eventName": "ActiveCellChangeEvent", "eventTime": 1699962681773, "eventInfo": {"cells": [{"id": "f5c73955-fc57-438c-9b81-2b197f5ce609", "index": 9}]}}, "notebookState": {"sessionID": "09ceefc2-82af-41d0-b806-a76369748020", "notebookPath": "alpha.ipynb", "notebookContent": null}}
```



---
File: /nebari-docs/docs/docs/how-tos/upgrade-kubernetes-version.md
---

---
id: kubernetes-version-upgrade
title: Upgrade cluster's Kubernetes version
description: A basic overview on how to upgrade your cluster Kubernetes version
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

## Overview

Nebari runs on Kubernetes under the hood, and as administrators of this Kubernetes cluster, one of the maintenance tasks required from time to time is upgrading the version of your Kubernetes cluster. Each of the different cloud providers that Nebari can run on, has their own release cycle and support windows for their flavor of Kubernetes. That said, they tend to follow the [official Kubernetes release cycle](https://kubernetes.io/releases/).

The Nebari development team tries to stay ahead of this by supporting the latest version when possible. However, given that many Kubernetes releases come with a set of deprecations that potentially affect Nebari and downstream plugins, there is an enforced [`HIGHEST_SUPPORTED_K8S_VERSION`](https://github.com/nebari-dev/nebari/blob/main/src/_nebari/constants.py#L11) allowed.

:::note
This `HIGHEST_SUPPORTED_K8S_VERSION` is, at times, a minor version or two behind the officially released Kubernetes version.
:::

Many cloud providers enable users to automatically upgrade their Kubernetes cluster (control plane), however given the potential for deprecations and other changes, Nebari deployed to these clouds has this feature automatically disabled.

This upgrade process bumps the version of the control plane along with all node groups.

:::warning

1. Upgrading the kubernetes version of the node groups will cause downtime so please plan accordingly. We also recommend [backing up your data](./manual-backup.md) before starting this upgrade process.
2. Kubernetes versions can only be upgraded one minor version at a time. If you're running on 1.24, and need to upgrade to 1.26, you will first need to upgrade to 1.25.
3. Downgrading to a lower version of Kubernetes is dangerous and we strongly advise against it!

:::

<Tabs>

<TabItem label="GCP" value="gcp" default="true">

Google Kubernetes Engine (GKE) cut their own platform specific version of Kubernetes that usually look something like: `1.26.7-gke.500`; this corresponds to a Kubernetes version of `1.26.7`.

You can list the supported GKE Kubernetes versions by running the following `gcloud` command:

```bash
gcloud container get-server-config --region <region>
```

We recommend selecting a `validVersion` from the `STABLE` channel:

```bash
channels:
- channel: RAPID
  defaultVersion: 1.27.4-gke.900
  validVersions:
  - 1.28.1-gke.201
  ...
- channel: REGULAR
  defaultVersion: 1.27.3-gke.100
  validVersions:
  - 1.27.4-gke.900
  - 1.27.3-gke.1700
  ...
- channel: STABLE
  defaultVersion: 1.27.3-gke.100
  validVersions:
  - 1.27.4-gke.900
  - 1.27.3-gke.100
  - 1.26.7-gke.500
  ...
  - 1.24.15-gke.1700
  - 1.24.14-gke.2700
```

To upgrade your GKE cluster, update the `google_cloud_platform.kubernetes_version` field in your `nebari-config.yaml` to match one of these GKE Kubernetes versions. Then run `nebari deploy` to apply these changes. This deployment process might take as long as 30 minutes.

:::info
You will get a validation error if you try to select a Kubernetes version that is unsupported by GKE or a version higher than [`HIGHEST_SUPPORTED_K8S_VERSION`][highest-supported-k8s].
:::

Then repeat the above process one minor version at a time. You will get a similar error otherwise:

```bash
[terraform]: │ Error: googleapi: Error 400: Master cannot be upgraded to "1.26.7-gke.500": cannot upgrade the master more than a minor version at a time.
```

For more information about GKE upgrades, please refer to the [GKE documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster).

</TabItem>

<TabItem label="AWS" value="aws">

The AWS Elastic Kubernetes Service (EKS) only requires that you supply the major and minor version of Kubernetes that you want. To specify Kubernetes version `1.26.7`, update the `amazon_web_services.kubernetes_version` to `1.26`. Then run `nebari deploy` to apply these changes. This deployment process might take as long as 30 minutes.

:::info
You will get a validation error if you try to select a Kubernetes version that is unsupported by EKS or a version higher than [`HIGHEST_SUPPORTED_K8S_VERSION`][highest-supported-k8s].
:::

In AWS, upgrading EKS will upgrade the control plane components but the **node groups will need to be upgraded manually**.

In the AWS console, navigate to EKS and click on the name of your Kubernetes cluster (format will be `{project-name}-{namespace}`). In the 'Compute' tab, scroll down to "Node Groups". Any node groups which are behind will have an "Update Now" button by the "AMI release version" column values. Click "Update Now" for each. Each update may take 15 or more minutes depending on how many workloads need to be migrated, but they can be run simultaneously.

Then repeat the above process one minor version at a time.

For more information about EKS upgrades, please refer to the [EKS documentation](https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html).

</TabItem>

<TabItem label="Azure" value="azure">

The Azure Kubernetes Service (ASK) requires that you specify the major, minor, and patch version you would like to use. To specify a Kubernetes version update the `azure.kubernetes_version` to `1.26.7` (or the version you need to upgrade to). Then run `nebari deploy` to apply these changes. This deployment process might take as long as 30 minutes.

:::info
You will get a validation error if you try to select a Kubernetes version that is unsupported by ASK or a version higher than [`HIGHEST_SUPPORTED_K8S_VERSION`][highest-supported-k8s].
:::

Then repeat the above process one minor version at a time.

For more information about ASK upgrade, please refer to the [ASK documentation](https://learn.microsoft.com/en-us/azure/ask/upgrade-cluster?tabs=azure-cli).

</TabItem>

</Tabs>

<!-- Reusable links -->

[highest-supported-k8s]: https://github.com/nebari-dev/nebari/blob/91792952b67074b5c15c3b4009bde5926ca4ec6b/src/_nebari/constants.py#L11



---
File: /nebari-docs/docs/docs/how-tos/use-gpus.mdx
---

---
id: use-gpus
title: Use GPUs on Nebari
description: Overview of using GPUs on Nebari including server setup, environment setup, and validation.
---

# Using GPUs on Nebari
## Introduction
Overview of using GPUs on Nebari including server setup, environment setup, and validation.

## 1. Starting a GPU server

  Follow Steps 1 to 3 in the [Authenticate and launch JupyterLab][login-with-keycloak] tutorial. The UI will show a list of profiles (a.k.a, instances, servers, or machines).

  ![Nebari select profile](/img/how-tos/nebari_select_profile.png)
  Your administrator pre-configures these options, as described in [Profile Configuration documentation][profile-configuration].

  Select an appropriate GPU Instance and click "Start".

  ### Understanding GPU setup on the server.
  The following steps describe how to get CUDA-related information from the server.
  1. Once your server starts, it will redirect you to a JupyterLab home page.
  2. Click on the **"Terminal"** icon.
  3. Run the command `nvidia-smi`. The top right corner of the command's output should have the highest supported driver.
      ![nvidia-smi-output](/img/how-tos/nvidia-smi-output.png)

  If you get the error `nvidia-smi: command not found`, you are most likely on a non-GPU server. Shutdown your server, and start up a GPU-enabled server.

  **Compatible environments for this server must contain CUDA versions *below* the GPU server version. For example, the server in this case is on 12.4. All environments used on this server must contain packages build with `CUDA<=12.4`.**

## 2. Creating environments

  By default, `conda-store` will build CPU-compatible packages. To build GPU-compatible packages, we do the following.
  ### Build a GPU-compatible environment
  By default, `conda-store` will build CPU-compatible packages. To build GPU-compatible packages, we have two options:
  1. **Create the environment specification using `CONDA_OVERRIDE_CUDA` (recommended approach)**:

    Conda-store provides an alternate mechanism to enable GPU environments via the setting of an environment variable as explained in the [conda-store docs](https://conda.store/conda-store-ui/tutorials/create-envs#set-environment-variables).
    While creating a new config, click on the `**GUI <-> YAML**` Toggle to edit yaml config.
    ```
    channels:
      - pytorch
      - conda-forge
    dependencies:
      - pytorch
      - ipykernel
    variables:
      CONDA_OVERRIDE_CUDA: "12.1"
    ```
    Alternatively, you can configure the same config using the UI.

    Add the `CONDA_OVERRIDE_CUDA` override to the variables section to tell conda-store to build a GPU-compatible environment.

:::note
At the time of writing this document, the latest CUDA version was showing as `12.1`. Please follow the steps below to determine the latest override value for the `CONDA_OVERRIDE_CUDA` environment variable.

Please ensure that your choice from PyTorch documentation is not greater than the highest supported version in the `nvidia-smi` output (captured above).
:::

  2. **Create the environment specification based on recommendations from the PyTorch documentation**:
    You can check [PyTorch documentation](https://pytorch.org/get-started/locally/) to get a quick list of the necessary CUDA-specific packages.
    Select the following options to get the latest CUDA version:
      - PyTorch Build = Stable
      - Your OS          = Linux
      - Package          = Conda
      - Language         = Python
      - Compute Platform = 12.1 (Select the version that is less than or equal to the `nvidia-smi` output (see above) on your server)

    ![pytorch-linux-conda-version](/img/how-tos/pytorch-linux-conda-version.png)

    The command `conda install` from above is:
    ```
    conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
    ```
    The corresponding yaml config would be:
    ```
    channels:
      - pytorch
      - nvidia
      - conda-forge
    dependencies:
      - pytorch
      - pytorch-cuda==12.1
      - torchvision
      - torchaudio
      - ipykernel
    variables: {}
    ```
    :::note
    The order of the channels is respected by conda, so keep pytorch at the top, then nvidia, then conda-forge.

    You can use `**GUI <-> YAML**` Toggle to edit the config.


## 3. Validating the setup
  You can check that your GPU server is compatible with your conda environment by opening a Jupyter Notebook, loading the environment, and running the following code:
  ```
  import torch
  print(f"GPU available: {torch.cuda.is_available()}")
  print(f"Number of GPUs available: {torch.cuda.device_count()}")
  print(f"ID of current GPU: {torch.cuda.current_device()}")
  print(f"Name of first GPU: {torch.cuda.get_device_name(0)}")
  ```
  Your output should look something like this:

  ![jupyter-notebook-command-output](/img/how-tos/pytorch-cuda-check.png)

<!-- Internal links -->
[profile-configuration]: /docs/explanations/profile-configuration
[login-with-keycloak]: /docs/tutorials/login-keycloak



---
File: /nebari-docs/docs/docs/how-tos/using-argo.md
---

---
id: using-argo
title: Automate your first workflow with Argo
description: Argo workflow management
---

# Automate workflows with Argo

[Argo Workflows](https://argoproj.github.io/workflows) is an open source container-native
workflow engine for orchestrating parallel jobs on Kubernetes. In other words,
Argo helps you run a sequence of tasks or functions without you having to be
present (it will manage the server resources for you). Argo workflows
comes enabled by default with Nebari deployments.

Access control for Argo on Nebari is done through Keycloak user groups. All
users in the `admin` or `developer` groups have access to Argo.

:::note
Also see the [Set up Argo Workflows documentation](/docs/how-tos/setup-argo).
:::

## Access the Argo Server

If Argo Workflows is enabled, users can access Argo Workflows UI at:
`your-nebari-domain.com/argo`. Log in via Keycloak with your usual credentials.

You can also download the
[Argo CLI](https://github.com/argoproj/argo-workflows/releases) if you prefer
a command line experience.

## Introduction to the Argo UI

Navigate to the Argo UI at `your-nebari-domain.com/argo`.

![Argo Server Landing Page](/img/how-tos/argo_landing_page.png)

From this page, you can see all the Argo servers currently running for each
workflow.

For kubernetes deployments, it important to note that these are
active pods. The two workflows shown in the UI above indicate that the workflows
are complete (the green check), but that the server is still running.

:::warning
We highly recommend setting the default timeout, otherwise the Argo pods will not
be culled on their own!
:::

You can click on each individual workflow to see the DAG and details for each
step in the workflow.

![Argo workflow detail](/img/how-tos/argo_workflow_details.png)

## Submit a workflow

You can submit a workflow through the UI by clicking "+ SUBMIT NEW WORKFLOW" on
the landing page. Argo offers a template for the workflow yaml format.

![Argo UI submit new workflow](/img/how-tos/argo_submit_new_workflow.png)

Click `+ CREATE` when you're ready to submit. The yaml format is not the only
option for generating workflows. Argo also allows you to create workflows via
python. More information on how to generate these specifications will follow.

## Submit a workflow via Argo CLI

You can also submit or manage workflows via the Argo CLI. The Argo CLI can be
downloaded from the
[Argo Releases](https://github.com/argoproj/argo-workflows/releases) page.

You can submit a workflow through the CLI using `argo submit my-workflow.yaml`.

The `argo list` command will list all the running workflows.

If you've just submitted a workflow and you want to check on it, you can run
`argo get @latest` to get the latest submitted workflow.

You can also access the logs for a workflow using
`argo logs -n workflow_name @latest`.

For more information on Argo workflows via the UI or the CLI, you can visit the
[Argo docs](https://argoproj.github.io/argo-workflows/workflow-concepts/).

[Hera](https://hera-workflows.readthedocs.io/) is a framework for building and
submitting Argo workflows in Python. Learn more in the [Argo Workflows walkthrough tutorial](/docs/tutorials/argo-workflows-walkthrough).

## Access your Nebari environments and file system while on an Argo pod (BETA)

Once you move beyond the "Hello World" Argo examples, you may realize that the
conda environments and the persistent storage you have on Nebari would be
really useful in your temporary Argo pods. Lucky for you, we've solved that
problem for you!

Nebari comes with [Nebari Workflow Controller (BETA)](https://github.com/nebari-dev/nebari-workflow-controller), abbreviated as NWC,
which transfers the user's environment variables, home and shared directories,
docker image, and available conda environments to the server where the Workflow
is running. Users can then run a script that loads and saves from their home
directory with a particular conda environment.

All of these things are enabled when users add the `jupyterflow-override` label
to their workflow as in this example using Hera:

```python
from hera.workflows import Workflow
Workflow(
    ...
    labels = {`jupyterflow-override`: 'true'},
)
```

Behind the scenes, NWC will override a portion of the workflow spec, mount
directories, etc. The label can be added to the Workflow in a kubernetes
manifest, via Hera, the Argo CLI, or via the Argo Server Web UI.

:::note
This feature requires that you have a Jupyter user pod running when the "jupyterflow-override" workflow is submitted. The workflow will not be created if you don't have a Jupyter user pod running.
:::

### YAML Example

```
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: jupyterflow-override-example
  namespace: dev  # replace with your deployment namespace if needed
  labels:
    jupyterflow-override: 'true'
spec:
  entrypoint: print-hello-world
  templates:
    - name: print-hello-world
      container:
        name: main
        image: any-image:any-tag  # will be overridden with the jupyter user pod image
        command:
          - sh
          - "-c"
        args:
          - "conda run -n nebari-git-dask python -c \"print('hello world')\" >> output_file_in_home_dir.txt"
```

The command starting with "conda run" runs the python print statement with the nebari-git-dask conda environment and writes stdout to the user's home directory, but could just as easily run a script in the user's home directory. The example above is for a Workflow, but the same process will work with [CronWorkflows](https://argoproj.github.io/argo-workflows/cron-workflows/) as well. In that case, the Jupyter user pod only needs to be running when the CronWorkflow is submitted, not each time the CronWorkflow starts a Workflow.

The jupyterflow-override feature is in beta so please [leave some feedback](https://github.com/nebari-dev/nebari-workflow-controller/discussions) and [report any issues](https://github.com/nebari-dev/nebari-workflow-controller/issues).

## Additional Argo Workflows Resources

Refer to the [Argo documentation](https://argoproj.github.io/argo-workflows/) for further details on Argo Workflows.

## Next Steps

Now that you have had an introduction, check out the [more detailed tutorial](/tutorials/argo-workflows-walkthrough.md) on
Argo for some practical examples!



---
File: /nebari-docs/docs/docs/how-tos/using-vscode.md
---

# Using Visual Studio (VS) Code

## Using VS Code as your development environment

VS Code can be used as an IDE (integrated development environment) which
provides helpful tooling (including debugging) to assist developers in writing
code. It also has many other functions which non-developers may also find
useful such a text editing and markdown rendering.

:::warning
Your VS Code server does not automatically shut down when idle. To prevent excess resource usage, be sure to manually shut down your server. More details are available in this [FAQ for VS Code][faq-vscode].
:::

## Getting started

VS Code comes built-in with every installation of Nebari. To start, log in
to Nebari and spin up a JupyterLab instance.

Next, bring up the `New Launcher` window by clicking the `+` in the top left of
the screen. Now click on the VS Code logo on the Launcher window.

![JupyterLab Launcher window with VS Code](/img/tutorials/vscode_launcher.png)

You will now have been redirected to a new web browser page showing the VS
Code platform. If you're starting VS Code for the first time, you'll see a
Welcome Page with some helpful links and tips.

Feel free to explore!

![VS Code Welcome screen](/img/tutorials/vscode_welcome.png)

## VS Code components

On the far left, you'll see the `Activity Bar` in black. Also, on the left is
the `Explorer`. As you click on the items in the `Activity Bar`, the `Explorer`
items will update.

Let's review some of the most useful features.

### The Activity Bar components

The `Activity Bar` is where you'll go to switch between the main tools
available in VS Code. Below is a brief overview of the icons on the
`Activity Bar` (adding extensions may add additional icons to your menu).

| Icon                                                                       | Name           | Description                                                               |
| -------------------------------------------------------------------------- | -------------- | ------------------------------------------------------------------------- |
| ![VS Code hamburger button](/img/tutorials/vscode_hamburger.png)           | File Menu      | Like every other file menu - create files, run files, edit preferences... |
| ![VS Code files button](/img/tutorials/vscode_files.png)                   | File Explorer  | View list of files, navigate folder structures                            |
| ![VS Code search button](/img/tutorials/vscode_search.png)                 | Search         | Search for words in the contents of files                                 |
| ![VS Code source control button](/img/tutorials/vscode_source_control.png) | Source Control | Source Control Management (SCM) features (e.g. git)                       |
| ![VS Code debug button](/img/tutorials/vscode_debug.png)                   | Debug          | Run code using the debugger                                               |
| ![VS Code extensions button](/img/tutorials/vscode_extensions.png)         | Extensions     | Add plugins to extend VS Code functionality                               |

## File editing

Now that we have that out of the way, let's explore!

We'll start by clicking on the `File Explorer` icon. The `Explorer` sidebar now
is updated with our file system. In our case, this is our Nebari user root
directory.

One of the first things you'll notice here is that there are a lot of files
starting with the `.` character. This is particularly handy because JupyterLab
hides these files in its Explorer view.

Let's click on a file we all have, `.bashrc`. This file was created by Nebari
for us.

![VS Code bashrc file](/img/tutorials/vscode_bashrc.png)

We now have an `Editor` window in which we can modify the file. The default
VS Code preferences include an auto-save feature which will continually save
the files as soon as you stop typing edits.

## Adding extensions

Using the `Activity Bar`, navigate to the `Extensions`. The `Explorer` sidebar
now shows lists of Extensions, grouped by those installed by your admin, those
installed by you, and a list of "Popular" extensions you may want to try.
Through this interface we can also search the Marketplace for a particular
extension.

![VS Code extensions list](/img/tutorials/vscode_extensions_list.png)

The Python extension is at the top of the list in our example (rightly so!),
but if you don't see it here, you can search for it.

If you click on the Python extension, you'll see additional information about
this extension in the main window. This extension provides some extra tooling
around Python. It will allow us to select a Python environment and run and
debug code right inside of VS Code. Go ahead and click `Install`. It should
only take a few seconds to install.

Now let's run some code!

## Running Python code

In the `Explorer` pane, selected the `New File` icon:

![VS Code select new file](/img/tutorials/vscode_new_file.png)

You can name it anything you'd like, here we've named it `example.py`. The
`*.py` extension let's VS Code know that it's a Python file, and you can set up
some automatic linting tasks through Preferences if you'd like.

Add some sample code to your file:

![VS Code new file contents](/img/tutorials/vscode_new_file_contents.png)

We also see the VS Code `Terminal` in the previous image. This can be
opened with `File -> Terminal -> New Terminal`.

Before we run our file, we need to select what Conda environment it should run
inside.

![VS Code select conda env](/img/tutorials/vscode_select_env.png)

Now we can run our code through the VS Code UI by selecting either
`Run Python File` or `Debug Python File` in the top right.

![VS Code run or debug env](/img/tutorials/vscode_run_debug_buttons.png)

We can see output in the terminal:

![VS Code view output](/img/tutorials/vscode_output.png)

As another alternative, we could run code directly from the terminal as well.

## Conclusion

We've covered the basic setup and run some arbitrary code. If you're curious
about more advanced features or have specific questions, you can always refer back
to the [VS Code Documentation](https://code.visualstudio.com/).

<!-- Internal links -->

[faq-vscode]: ../faq#why-does-my-vs-code-server-continue-to-run-even-after-ive-been-idle-for-a-long-time



---
File: /nebari-docs/docs/docs/references/container-sources.md
---

## Deploying and Running Nebari from a Private Container Repository

Nebari deploys and runs FOSS components as containers running in Kubernetes.
By default, Nebari sources each container from the container's respective public repository, typically `docker.io` or `quay.io`.
This introduces supply-chain concerns for security-focused customers.

One solution to these supply-chain concerns is to deploy Nebari from private locally-mirrored containers:

- Create a controlled private container repository (e.g. ECR)
- Mirror all containers used by Nebari into this private container repository
- Use the `pre_bootstrap_command` mechanism in `nebari-config.yaml` to specify the mirrored container repo

Deploying Nebari in this fashion eliminates significant supply chain surface-area, but requires identifying all containers used by Nebari.

The following configurations demonstrate how to specify a private repo denoted by the string `[PRIVATE_REPO]`.

**Note:** Authorization tokens are used in the examples below. It is important for administrators to understand the expiration policy of these tokens, because the Nebari k8s cluster may in some cases need to **use these tokens to pull container images at any time during run-time operation**.

### Set ECR as default container registry mirror

```
amazon_web_services:
  node_groups:
    general:
      instance: m5.2xlarge
      launch_template:
        pre_bootstrap_command: |
            #!/bin/bash
            # Verify that IP forwarding is enabled for worker nodes, as is required for containerd
            if [[ $(sysctl net.ipv4.ip_forward | grep "net.ipv4.ip_forward = 1") ]]; then echo "net.ipv4.ip_forward is on"; else sysctl -w net.ipv4.ip_forward=1; fi
            # Set ECR as default container registry mirror
            mkdir -p /etc/containerd/certs.d/_default
            ECR_TOKEN="$(aws ecr get-login-password --region us-east-1)"
            BASIC_AUTH="$(echo -n "AWS:$ECR_TOKEN" | base64 -w 0)"
            cat <<-EOT > /etc/containerd/certs.d/_default/hosts.toml
            [host."https://[PRIVATE_REPO].dkr.ecr.us-east-1.amazonaws.com"]
              capabilities = ["pull", "resolve"]
              [host."https://[PRIVATE_REPO].dkr.ecr.us-east-1.amazonaws.com".header]
                authorization = "Basic $BASIC_AUTH"
            EOT

```

### Set GitLab CR as default container registry mirror

```
# Set GitLab CR as default container registry mirror in hosts.toml;
# must have override_path set if project/group names don't match upstream container
amazon_web_services:
  node_groups:
    general:
      instance: m5.2xlarge
      launch_template:
        pre_bootstrap_command: |
            #!/bin/bash
            # Verify that IP forwarding is enabled for worker nodes, as is required for containerd
            if [[ $(sysctl net.ipv4.ip_forward | grep "net.ipv4.ip_forward = 1") ]]; then echo "net.ipv4.ip_forward is on"; else sysctl -w net.ipv4.ip_forward=1; fi
            # Set default container registry mirror in hosts.toml; must have override_path set if project/group names don't match upstream container
            CONTAINER_REGISTRY_URL="[PRIVATE_REPO]"
            CONTAINER_REGISTRY_USERNAME="[username]"
            CONTAINER_REGISTRY_TOKEN="[token]"
            CONTAINER_REGISTRY_GROUP=as-nebari
            CONTAINER_REGISTRY_PROJECT=nebari-test
            mkdir -p /etc/containerd/certs.d/_default
            cat <<-EOT > /etc/containerd/certs.d/_default/hosts.toml
            [host."https://$CONTAINER_REGISTRY_URL/v2/$CONTAINER_REGISTRY_GROUP/$CONTAINER_REGISTRY_PROJECT"]
              override_path = true
              capabilities = ["pull", "resolve"]
            EOT

            # Set containerd registry config auth in config.d .toml import dir
            mkdir -p /etc/containerd/config.d
            cat <<EOT | sudo tee /etc/containerd/config.d/config-import.toml
            version = 2
            [plugins."io.containerd.grpc.v1.cri".registry]
              config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
              [plugins."io.containerd.grpc.v1.cri".registry.auths]
              [plugins."io.containerd.grpc.v1.cri".registry.configs]
                [plugins."io.containerd.grpc.v1.cri".registry.configs."$CONTAINER_REGISTRY_URL".auth]
                  username = "$CONTAINER_REGISTRY_USERNAME"
                  password = "$CONTAINER_REGISTRY_TOKEN"
            EOT
```

### Set GitLab CR as default container registry mirror, with custom Client SSL/TLS Certs

```
# must have override_path set if project/group names don't match upstream container
# Also add/set GitLab Client SSL/TLS Certificate for Containerd
amazon_web_services:
  node_groups:
    general:
      instance: m5.2xlarge
      launch_template:
        pre_bootstrap_command: |
            #!/bin/bash
            # Verify that IP forwarding is enabled for worker nodes, as is required for containerd
            if [[ $(sysctl net.ipv4.ip_forward | grep "net.ipv4.ip_forward = 1") ]]; then echo "net.ipv4.ip_forward is on"; else sysctl -w net.ipv4.ip_forward=1; fi
            # Set default container registry mirror in hosts.toml; must have override_path set if project/group names don't match upstream container
            CONTAINER_REGISTRY_URL="[PRIVATE_REPO]"
            CONTAINER_REGISTRY_USERNAME="[username]"
            CONTAINER_REGISTRY_TOKEN="[token]"
            CONTAINER_REGISTRY_GROUP=as-nebari
            CONTAINER_REGISTRY_PROJECT=nebari-test
            mkdir -p /etc/containerd/certs.d/_default
            cat <<-EOT > /etc/containerd/certs.d/_default/hosts.toml
            [host."https://$CONTAINER_REGISTRY_URL/v2/$CONTAINER_REGISTRY_GROUP/$CONTAINER_REGISTRY_PROJECT"]
              override_path = true
              capabilities = ["pull", "resolve"]
              client = ["/etc/containerd/certs.d/$CONTAINER_REGISTRY_URL/client.pem"]
            EOT

            # Set containerd registry config auth in config.d .toml import dir
            mkdir -p /etc/containerd/config.d
            cat <<EOT | sudo tee /etc/containerd/config.d/config-import.toml
            version = 2
            [plugins."io.containerd.grpc.v1.cri".registry]
              config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
              [plugins."io.containerd.grpc.v1.cri".registry.auths]
              [plugins."io.containerd.grpc.v1.cri".registry.configs]
                [plugins."io.containerd.grpc.v1.cri".registry.configs."$CONTAINER_REGISTRY_URL".auth]
                  username = "$CONTAINER_REGISTRY_USERNAME"
                  password = "$CONTAINER_REGISTRY_TOKEN"
            EOT

            # Add client key/cert to containerd
            mkdir -p /etc/containerd/certs.d/$CONTAINER_REGISTRY_URL
            cat <<-EOT >> /etc/containerd/certs.d/$CONTAINER_REGISTRY_URL/client.pem
            -----BEGIN CERTIFICATE-----
            XzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzZx
            ZxyzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzXz
            -----END CERTIFICATE-----
            -----BEGIN PRIVATE KEY-----
            XzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzZx
            ZxyzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzXz
            -----END PRIVATE KEY-----
            EOT
```



---
File: /nebari-docs/docs/docs/references/enhanced-security.md
---

## Nebari Security Considerations

The security of _AWS Nebari_ deployments can be enhanced through the following deployment configuration options in `nebari-config.yaml`:

- **Explicit definition of container sources**
  This option allows for the use of locally mirrored, security-hardened, or otherwise customized container images in place of the containers used by default.
  See: [container-sources](container-sources.md)

- **Installation of custom SSL certificate(s) into EKS hosts**
  Install private certificates used by (e.g.) in-line content inspection engines which re-encrypt traffic.

```
# Add client certificate to CA trust on node
amazon_web_services:
  node_groups:
    general:
      instance: m5.2xlarge
      launch_template:
        pre_bootstrap_command: |
            #!/bin/bash
            cat <<-EOT >> /etc/pki/ca-trust/source/anchors/client.pem
            -----BEGIN CERTIFICATE-----
            XzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzZx
            ZxyzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxxzxzxzxzxzxzxzxzxzxxzxzXz
            -----END CERTIFICATE-----
            EOT
            sudo update-ca-trust extract
```

- **Private EKS endpoint configuration**
  Mirrors the corresponding AWS console option, which routes all EKS traffic within the VPC.

```
  amazon_web_services:
    eks_endpoint_access: private # valid values: [public, private, public_and_private]
```

- **Deploy into existing subnets**
  Instructs Nebari to be deployed into existing subnets, rather than creating its own new subnets.
  An advantage of deploying to existing subnets is the ability to use private subnets. Note that the **ingress load-balancer-annotation** must be set appropriately based on the type (private or public) of subnet.

```
existing_subnet_ids:
    - subnet-0123456789abcdef
    - subnet-abcdef0123456789
  existing_security_group_id: sg-0123456789abcdef
ingress:
  terraform_overrides:
    load-balancer-annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      # Ensure the subnet IDs are also set below
      service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-0123456789abcdef,subnet-abcdef0123456789"
```

- **Use existing SSL certificate**
  Instructs Nebari to use the SSL certificate specified by `[k8s-custom-secret-name]`

```
certificate:
  type: existing
  secret_name: [k8s-custom-secret-name]
```



---
File: /nebari-docs/docs/docs/references/index.mdx
---

import DocCardList from '@theme/DocCardList';
import {useCurrentSidebarCategory} from '@docusaurus/theme-common';

# Reference guides

<div align="center">
  <img
    src="/img/welcome/references-icon.svg"
    width="30%"
    style={{border: "none", paddingBottom: 10 + "px", boxShadow: "none"}}
  />
</div>

Technical descriptions of how Nebari works.

- [Enhanced Security](enhanced-security.md) - Nebari security configuration guide
- [Local Container Repo](container-sources.md) - Deploying Nebari from a Local Container Repo
<DocCardList items={useCurrentSidebarCategory().items}/>



---
File: /nebari-docs/docs/docs/references/personas.md
---

---
# Display h2 to h3 headings
toc_min_heading_level: 2
toc_max_heading_level: 3
---

# Personas

There are a variety of people which may interact with Nebari deployments. Below is a list of fabricated personas intended to cover most people who interact with Nebari.

## User groups

Each persona is placed in a Nebari user group:

**End users** are the folks who are doing their daily work on Nebari. They are using Nebari as a platform to complete a variety of tasks from software development to dashboard creation to machine learning training and evaluation. Our end users are [Alia](#alia), [Noor](#noor), [Robin](#robin), and [Skyler](#skyler).

**Super users**, like end users may also be using Nebari for their daily work, but they do so with elevated privileges. These folks are in charge of managing teams of people and as such may be managing environments for their team. Our super users are [Blake](#blake), [Enzo](#enzo), [Jacob](#jacob), and [Sam](#sam).

**Customer** represents the person at an organization making the high level decision to adopt Nebari as a platform for their employees. Our customer is [Jordan](#jordan).

**Admin** represents the people who are in charge of deploying Nebari. They are responsible for maintaining and running the day to day maintenance. This may include cloud account management, conda-store maintenance, kubernetes cluster management and adding/removing end users from the deployment. Our admin is [Taylor](#taylor)

## Personas

---

### Alia

<table>
    <tr>
        <td style={{border: 'none'}}>
            <p>Age bracket: 25-35</p>
            <p>Role: Data scientist </p>
            <p>Reports to: Head of Data Science</p>
            <p>Nebari user group: end user</p>
        </td>
        <td style={{border: 'none'}}>
            <img src="/img/references/alia.png" alt="cartoon image of persona alia" style={{ border: 'none', 'background-color' : 'var(--ifm-color-background-3)', height: 300}}  />
        </td>
    </tr>
</table>

#### 📣 Story

- As a Data scientist I want to investigate, extract and report meaningful insights with my organisation’s data.
- As a DS I want to build self-serving internal data products to make data simple within the company.

#### :wrench: Software tools

- Python - NumPy, Pandas, scikit-learn, etc.
- Airflow
- GCP buckets
- SQL
- Data viz and dashboarding tools like Plotly, Panel, Altair...
- VSCode, Jupyter notebooks

#### 🌮 Core needs

N/A

#### 🐛 Pain points or biggest challenges

- Poor data quality and quantity
- High friction in collaborative projects

- Ability to experiment with new methods, libraries, and datasets in an efficient and quick way
- Access to multiple data sources with minimal friction
- Clear communication channels with software engineering, infrastructure, and other departments
- Need to communicate results to non-technical people

---

### Blake

<table>
    <tr>
        <td style={{border: 'none'}}>
            <p>Age bracket: 25-35 </p>
            <p>Role: Software Engineer/ ML Engineer </p>
            <p>Reports to: Head of Software Engineering </p>
            <p>Nebari user group: super user</p>
        </td>
        <td style={{border: 'none'}}>
            <img src="/img/references/blake.png" alt="cartoon image of persona blake" style={{ border: 'none', 'background-color' : 'var(--ifm-color-background-3)', height: 300}}  />
        </td>
    </tr>
</table>

#### 📣 Story

- Nebari is a place I go to do my everyday programming work. I use conda envs, jupyter, and write code in vscode. I deploy dashboards with cds dashboards. I create packages and I need to pip install the dev versions into my environment.
- As a SWE I want to be able to collaborate with the Data and Engineering teams to build new prototypes for products

#### :wrench: Software tools

- VSCode
- Jupyter notebooks
- conda, pip
- Python and stack
- Plotly, Altair, etc.

#### 🐛 Pain points or biggest challenges

N/A

#### 🌮 Core needs

- Need to be able to create prototypes efficiently and run experiments with product and data teams
- ability to ship products without being held up in the process
- ability to manage my environment and tooling

---

### Enzo

<table>
    <tr>
        <td style={{border: 'none'}}>
            <p>Age bracket: 30-40</p>
            <p>Role: Staff Machine learning engineer at a startup </p>
            <p>Reports to: Head of Software Engineering</p>
            <p>Nebari user group: super user</p>
        </td>
        <td style={{border: 'none'}}>
            <img src="/img/references/enzo.png" alt="cartoon image of persona enzo" style={{ border: 'none', 'background-color' : 'var(--ifm-color-background-3)', height: 300}}  />
        </td>
    </tr>
</table>

#### 📣 Story

- As an ML Engineer in a startup, I have good modeling and coding skills, but I want to build MLOps pipelines as easily as possible, without having to learn DevOps first, so that I can efficiently test and productionize my models.

#### :wrench: Software tools

- Python, PyTorch
- Git, GitHub
- AWS, GCP
- Argo
- DVC, ClearML, Tensorboard
- CI/CD platform
- K8s, Dask

#### 🐛 Pain points or biggest challenges

- Lack of DevOps experience makes pipeline implementation difficult, time consuming, and error prone.
- Limited high-level access to pipeline tools means that they are essentially black boxes to me (can’t see logs, not sure what’s happening when something breaks)

#### 🌮 Core needs

- Safe and efficient access to infrastructure without breaking things
- Elevated privileges to experiment with different configurations and for transparency into background tasks
- Documentation that shows how to implement common pipeline configurations
- Templates/base configurations with several different scenarios

---

### Jacob

<table>
    <tr>
        <td style={{border: 'none'}}>
            <p>Age bracket: 30-40  </p>
            <p>Role: Staff Machine learning engineer  </p>
            <p>Reports to: Head of Software Engineering</p>
            <p>Nebari user group: super user</p>
        </td>
        <td style={{border: 'none'}}>
            <img src="/img/references/jacob.png" alt="cartoon image of persona jacob" style={{ border: 'none', 'background-color' : 'var(--ifm-color-background-3)', height: 300}}  />
        </td>
    </tr>
</table>

#### 📣 Story

- As an ML Engineer with good DevOps skills and not much time, I want to build MLOps pipelines efficiently so that I don’t waste time on tedious tasks.

#### :wrench: Software tools

- Python, PyTorch
- Git, GitHub
- AWS, GCP
- Argo
- DVC, ClearML, Tensorboard
- CI/CD platform
- K8s, Dask

#### 🐛 Pain points or biggest challenges

- Rigid pipelining tools limit flexibility to build pipelines that meet my specific constraints.
- Limited or incomplete documentation (just hello world examples) doesn’t give me enough information to build the complex configuration I need without a lot of trial and error testing.

#### 🌮 Core needs

- Efficient access to infrastructure
- Elevated privileges to experiment with different configurations and for transparency into background tasks
- Documentation that shows how to implement common pipeline configurations
- Templates/base configurations with several different scenarios
- Flexible pipelining platform that allows experimentation and custom solutions

---

### Jordan

<table>
    <tr>
        <td style={{border: 'none'}}>
            <p>Age bracket: 45-60</p>
            <p>Role: CEO</p>
            <p>Reports to: &mdash; </p>
            <p>Nebari user group: customer</p>
        </td>
        <td style={{border: 'none'}}>
            <img src="/img/references/jordan.png" alt="cartoon image of persona jordan" style={{ border: 'none', 'background-color' : 'var(--ifm-color-background-3)', height: 300}}  />
        </td>
    </tr>
</table>

#### 📣 Story

- As a CEO I am interested in knowing how the data teams are helping us improve our processes and revenue streams.
- I don’t know anything about programming or jupyter. As the CEO I want to click on a button or email, sign in, and see informative dashboards and metrics.
- I want to have a clear breakdown of costs from my departments.

#### :wrench: Software tools

- Spreadsheets
- Email
- Data dashboards

#### 🐛 Pain points or biggest challenges

- there is big churn in the industry right now - how can we easily bring up new hires to speed?

#### 🌮 Core needs

- low-barrier access to data insights and outputs
- easy to track budgets and costs
- please stop the fights on data and swe departments

---

### Noor

<table>
    <tr>
        <td style={{border: 'none'}}>
            <p>Age bracket: 16-25</p>
            <p>Role: Student Python Programmer  </p>
            <p>Reports to: &mdash; </p>
            <p>Nebari user group: end user</p>
        </td>
        <td style={{border: 'none'}}>
            <img src="/img/references/noor.png" alt="cartoon image of persona noor" style={{ border: 'none', 'background-color' : 'var(--ifm-color-background-3)', height: 300}}  />
        </td>
    </tr>
</table>

#### 📣 Story

- As a student, I want to learn how to use Python to get a job in software development after I graduate.
- I’m learning how to code. I know I need an environment, but someone else creates and manages them. I never modify an environment myself (either through conda or pip)

#### :wrench: Software tools

- Python
- VScode and Jupyter notebooks
- conda, pip

#### 🌮 Core needs

- simple and intuitive platform
- easy sharing and collaboration
- ability to experiment without worrying about “breaking things”

#### 🐛 Pain points or biggest challenges

- I do not know how to share my work with friends and peers

---

### Robin

<table>
    <tr>
        <td style={{border: 'none'}}>
            <p>Age bracket: 45-55 </p>
            <p>Role: Head of Data Science  </p>
            <p>Reports to: CTO</p>
            <p>Nebari user group: end user</p>
        </td>
        <td style={{border: 'none'}}>
            <img src="/img/references/robin.png" alt="cartoon image of persona robin" style={{ border: 'none', 'background-color' : 'var(--ifm-color-background-3)', height: 300}}  />
        </td>
    </tr>
</table>

#### 📣 Story

- As the head of Data Science I want to make sure my team has all the tools and data needed to be successful in their job.
- As the HDS I want to have executive buy-in to grow the data practice in house to improve our internal processes and better serve our customers.
- As the HDS I want to not have to spend half of my budget in proprietary platforms.
- As the HDS I want to work with organisation leadership to help establish long-term vision, strategy and roadmap for one or more teams

#### :wrench: Software tools

- Git/ GitHub
- R and Python
- Dhasboarding and data viz tools
- Spark, SQL
- VSCode, Jupyter notebooks

#### 🌮 Core needs

- Low friction tooling to make my team more productive
- Foster collaboration, mentorship and peer-programming
- A platform that “just works”
- An easy way to share insights with non-technical executives and folks at the company

#### 🐛 Pain points or biggest challenges

- Lack of alignment across departments and lack of OKRs
- Often get caught on discussions between my team, software engineering, and infrastructure to move items to production
- Another resignation - need to offboard and onboard people seamlessly

---

### Sam

<table>
    <tr>
        <td style={{border: 'none'}}>
            <p>Age bracket: 30-40  </p>
            <p>Role: Research Data Scientist </p>
            <p>Reports to: Head of Data Science  </p>
            <p>Nebari user group: super user</p>
        </td>
        <td style={{border: 'none'}}>
            <img src="/img/references/sam.png" alt="cartoon image of persona sam" style={{ border: 'none', 'background-color' : 'var(--ifm-color-background-3)', height: 300}}  />
        </td>
    </tr>
</table>

#### 📣 Story

- As a research data scientist I want to use ML to measure and optimise the costs, performance, efficiency and reliability of our company’s infrastructure to deliver the best experience to our customers.
- As a research data scientist I want to implement and test out new approaches both on toy test tasks as well as on actual application scenarios.

#### :wrench: Software tools

- Python
- Machine Learning frameworks
- SQL
- Dask
- VSCode, Jupyter notebooks

#### 🌮 Core needs

- Collaboration with engineering, product and the rest of the DS team
- Ability to track experiments, models, data, accuracy, metrics reliably
- Ability to access scalable compute on-demand

#### 🐛 Pain points or biggest challenges

- Cannot scale the compute resources without having to go through infrastructure approval
- Not easy way to track experiments, data lineage, and workflow artefacts
- Do not understand git well enough

---

### Skyler

<table>
    <tr>
        <td style={{border: 'none'}}>
            <p>Age bracket: 35-45  </p>
            <p>Role: Staff Machine learning engineer  </p>
            <p>Reports to: Head of Software Engineering  </p>
            <p>Nebari user group: end user</p>
        </td>
        <td style={{border: 'none'}}>
            <img src="/img/references/skyler.png" alt="cartoon image of persona skyler" style={{ border: 'none', 'background-color' : 'var(--ifm-color-background-3)', height: 300}}  />
        </td>
    </tr>
</table>

#### 📣 Story

- As a ML engineer I want to work across teams and functions -- software engineers, ML engineers, data scientists, researchers, and product managers to understand user problems and build engineering solutions.
- As a ML Engineer I want to work with ML engineers and related teams to integrate ML components into the system and solve their performance issues

#### :wrench: Software tools

- Docker
- CI/CD platforms
- Python, Go, bash
- VSCode and Vim
- Flyte, Kubeflow
- GCP, Azure, AWS, DO
- Dask
- Git, GitHub, GitLab

#### 🌮 Core needs

- highly collaborative and extensible tooling and platform
- enforcement of best practices
- safe and efficient access to both the data and the infrastructure
- elevated privileges to experiment in “staging or pre-production” environments

#### 🐛 Pain points or biggest challenges

- Lack of best practices across other sides of the organisation makes my job really hard
- I am not sure if the models in production have bugs or the data is no longer accurate, need better monitoring tools

---

### Taylor

<table>
    <tr>
        <td style={{border: 'none'}}>
            <p>Age bracket: 30-45 </p>
            <p>Role: Sr. Dev Ops Engineer  </p>
            <p>Reports to: Head of Software Engineering  </p>
            <p>Nebari user group: admin</p>
        </td>
        <td style={{border: 'none'}}>
            <img src="/img/references/taylor.png" alt="cartoon image of persona taylor" style={{ border: 'none', 'background-color' : 'var(--ifm-color-background-3)', height: 300}}  />
        </td>
    </tr>
</table>

#### 📣 Story

- I maintain my org’s Nebari deployment. I make sure the infrastructure as code is maintained and that everyone has the conda environments or other resources they need.
- As a DevOps engineer I want to automate away as much of the day to day as possible - “Run By Robots” is the goal to minimise issues and make out team more efficient.
- Identify, respond to and collaborate with support and product teams to resolve production and customer issues and incidents.

#### :wrench: Software tools

- GCP, Azure, AWS, DO
- GNU/Linux
- Scripting - bash
- Terraform, Ansible, Docker...
- Kubernetes
- Grafana, Prometheus, Elastic

#### 🐛 Pain points or biggest challenges

- The organisation still relies on many manual operations
- There is so much technical debt - we can lose all of our infrastructure easily

#### 🌮 Core needs

- Rely on automation as much as needed
- Have a product and security first approach to data and SWE
- Have organisation wide best practices for monitoring, alerting and incident management
- Ensure the whole company adheres to best practices



---
File: /nebari-docs/docs/docs/references/RELEASE.md
---

# Release notes

_Contains description of Nebari releases._

<!-- Note:
The RELEASE.md file at the root of the Nebari codebase is the source of truth for all release notes.
If you want to update the release notes, open a PR against nebari-dev/nebari.
This file is copied to nebari-dev/nebari-docs using a GitHub Action. -->

---

## Release 2024.12.1 - December 13, 2024

> NOTE: Support for DigitalOcean has been removed in this release. If you plan to deploy Nebari on DigitalOcean, you first need to independently create a Kubernetes cluster and then use the `existing` deployment option.

### What's Changed

- Precommit typos by @blakerosenthal in https://github.com/nebari-dev/nebari/pull/2731
- fix typo in KubernetesCredentials by @blakerosenthal in https://github.com/nebari-dev/nebari/pull/2729
- handle branch rename from develop to main in github actions by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2748
- remove do integration test by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2765
- Remove old develop branch references after default branch renaming by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2769
- fix CICD issue with pre-commit action by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2775
- fix CHECK_URL in kuberhealthy checks to respect namespaces by @dcmcand in https://github.com/nebari-dev/nebari/pull/2779
- remove duplicate GCPPrivateClusterConfig class by @dcmcand in https://github.com/nebari-dev/nebari/pull/2786
- Fix hub variable for jupyterhub_dashboard by @kenafoster in https://github.com/nebari-dev/nebari/pull/2721
- Fix Pytest Tests failing on PRs updating src by @joneszc in https://github.com/nebari-dev/nebari/pull/2790
- Add ability to add overrides to jhub-apps config by @aktech in https://github.com/nebari-dev/nebari/pull/2754
- Remove leftover develop reference by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2792
- fix bug where check_immutable_fields throws error with old version of Nebari by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2796
- Fix immutable field validation error when a sub-schema is not Pydantic by @kenafoster in https://github.com/nebari-dev/nebari/pull/2797
- Address issue with AWS instance type schema by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2787
- add broken note by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2802
- Fix release notes formatting to restore docs syncing functionality by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2809
- Refactor role creation for upgrade command path by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2795
- add test workflow for upgrade by @pmeier in https://github.com/nebari-dev/nebari/pull/2780
- Add config option to enable the encryption of AWS EKS secrets by @joneszc in https://github.com/nebari-dev/nebari/pull/2788
- remove digital ocean tests by @dcmcand in https://github.com/nebari-dev/nebari/pull/2813
- Python3 13 upgrade dependencies by @dcmcand in https://github.com/nebari-dev/nebari/pull/2823
- Test support for Python 3.13 in CI by @aktech in https://github.com/nebari-dev/nebari/pull/2774
- remove unmaintained nix files by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2831
- allow passing X.XX or X.XX.XX as k8s versions by @dcmcand in https://github.com/nebari-dev/nebari/pull/2840
- Remove explicit branch inputs from cloud integration test workflows in GHA by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2837
- Allow overriding of keycloak root credentials for `2024.11.1` upgrade path by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2843
- Added security group rule descriptions by @jcbolling in https://github.com/nebari-dev/nebari/pull/2850
- Set `launch_template.ami_id` attrs to private by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2842
- attempt to address paramiko connection errors by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2811
- specify terraform registry for providers not in opentofu registry by @dcmcand in https://github.com/nebari-dev/nebari/pull/2852
- Disable AWS `launch_template` from nebari-config schema by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2856
- Remove Digital Ocean references by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2838
- Use tofu binary instead of terraform one by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2773
- Add 2024.11.1 release notes and bump version by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2859
- Disable `jupyterlab-jhub-apps` extension when jhub-apps is disabled by @krassowski in https://github.com/nebari-dev/nebari/pull/2804
- Validate instance types for GCP by @blakerosenthal in https://github.com/nebari-dev/nebari/pull/2730
- update gcp instance validation by @dcmcand in https://github.com/nebari-dev/nebari/pull/2875

### New Contributors

- @jcbolling made their first contribution in https://github.com/nebari-dev/nebari/pull/2850

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.11.1...2024.12.1

## Release 2024.11.1 - November 21, 2024 (Hotfix Release)

> NOTE: This hotfix addresses several major bugs identified in the 2024.9.1 release. For a detailed overview, please refer to the related discussion at #2798. Users should upgrade directly from 2024.7.1 to 2024.11.1.

## What's Changed

- fix `CHECK_URL` in kuberhealthy checks to respect namespaces by @dcmcand in https://github.com/nebari-dev/nebari/pull/2779
- fix bug where `check_immutable_fields` throws error with old version of Nebari by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2796
- Fix immutable field validation error when a sub-schema is not Pydantic by @kenafoster in https://github.com/nebari-dev/nebari/pull/2797
- Address issue with AWS instance type schema by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2787
- Add broken note by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2802
- Refactor role creation for upgrade command path by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2795
- Allow overriding of keycloak root credentials for 2024.11.1 upgrade path #2843
- Disable AWS `launch_template` from nebari-config schema #2856

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.9.1...2024.11.1

## Release 2024.9.1 - September 27, 2024 (Broken Release)

> WARNING: This release was later found to have unresolved issues described further in [issue 2798](https://github.com/nebari-dev/nebari/issues/2798). We have marked this release as broken on conda-forge and yanked it on PyPI. One of the bugs prevents any upgrade from 2024.9.1 to 2024.11.1. Users should skip this release entirely and upgrade directly from 2024.7.1 to 2024.11.1.

> WARNING: This release changes how group directories are mounted in JupyterLab pods: only groups with specific permissions will have their directories mounted. If you rely on custom group mounts, we strongly recommend running `nebari upgrade` before updating. This will prompt you to confirm how Nebari should handle your groups—either keep them mounted or allow unmounting. **No data will be lost**, and you can reverse this anytime.

### What's Changed

- Fix: KeyValueDict error when deploying to existing infrastructure by @oftheaxe in https://github.com/nebari-dev/nebari/pull/2560
- Remove unused AWS terraform modules by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2623
- Upgrade Hashicorp Vault action by @aktech in https://github.com/nebari-dev/nebari/pull/2616
- Pass `oauth_no_confirm=True` to jhub-apps by @krassowski in https://github.com/nebari-dev/nebari/pull/2631
- Use Rook Ceph for Jupyterhub and Conda Store drives by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2541
- Fix typo in guided init by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2635
- Action var tests off by @BrianCashProf in https://github.com/nebari-dev/nebari/pull/2632
- add a "moved" block to account for refactored terraform code without deleting/recreating NFS disks by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2639
- Use Helm Chart for JupyterHub 5.1.0 by @krassowski in https://github.com/nebari-dev/nebari/pull/2661
- Add a how to test section to PR template by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2659
- Support disallowed nebari config changes by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2660
- Fix converted init command in guided init by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2666
- Add initial uptime metrics by @dcmcand in https://github.com/nebari-dev/nebari/pull/2609
- Refactor and extend Playwright tests by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2644
- Remove Cypress remaining tests/files by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2672
- refactor jupyterhub user token retrieval within pytest by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2645
- add moved block to account for terraform changes on AWS only by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2673
- Refactor shared group mounting using RBAC by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2593
- Dashboard fix usage report by @kenafoster in https://github.com/nebari-dev/nebari/pull/2671
- only capture stdout not stdout+stderr when capture_output=True by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2704
- revert breaking change to azure deployment test by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2706
- Refactor GitOps approach prompt flow in guided init by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2269
- template the kustomization.yaml file by @dcmcand in https://github.com/nebari-dev/nebari/pull/2667
- Fix auto-provisioned GitHub repo description after guided init by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2708
- Add amazon_web_services configuration option to specify EKS cluster api server endpoint access setting by @joneszc in https://github.com/nebari-dev/nebari/pull/2618
- Use Google Auth and Cloud Python APIs instead of `gcloud` CLI by @swastik959 in https://github.com/nebari-dev/nebari/pull/2083
- fix broken links in README.md, SECURITY.md, and CONTRIBUTING.md by @blakerosenthal in https://github.com/nebari-dev/nebari/pull/2720
- add test for changing dicts and lists by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2724
- 2024.9.1 upgrade notes by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2726
- Add Support for AWS Launch Template Configuration by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2668
- Run terraform init before running terraform show by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2734
- Release Process Checklist Updates by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2727
- Test implicit aiohttp's TCP to HTTP connector change by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2741
- remove comments by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2743
- Deploy Rook Ceph Helm only when Ceph FS Needed by @kenafoster in https://github.com/nebari-dev/nebari/pull/2742
- fix group mounting paths by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2738
- Add compatibility prompt and notes for shared group mounting by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2739

### New Contributors

- @oftheaxe made their first contribution in https://github.com/nebari-dev/nebari/pull/2560
- @joneszc made their first contribution in https://github.com/nebari-dev/nebari/pull/2618
- @swastik959 made their first contribution in https://github.com/nebari-dev/nebari/pull/2083
- @blakerosenthal made their first contribution in https://github.com/nebari-dev/nebari/pull/2720

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.7.1...2024.9.1

## Release 2024.7.1 - August 8, 2024

> NOTE: Support for Digital Ocean deployments using CLI commands and related Terraform modules is being deprecated. Although Digital Ocean will no longer be directly supported in future releases, you can still deploy to Digital Ocean infrastructure using the current `existing` deployment option.

### What's Changed

- Enable authentication by default in jupyter-server by @krassowski in https://github.com/nebari-dev/nebari/pull/2288
- remove dns sleep by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2550
- Conda-store permissions v2 + load roles from keycloak by @aktech in https://github.com/nebari-dev/nebari/pull/2531
- Restrict public access and add bucket encryption using cmk by @dcmcand in https://github.com/nebari-dev/nebari/pull/2525
- Add overwrite to AWS coredns addon by @dcmcand in https://github.com/nebari-dev/nebari/pull/2538
- Add a default roles at initialisation by @aktech in https://github.com/nebari-dev/nebari/pull/2546
- Hide gallery section if no exhibits are configured by @krassowski in https://github.com/nebari-dev/nebari/pull/2549
- Add note about ~/.bash_profile by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2575
- Expose jupyterlab-gallery branch and depth options by @krassowski in https://github.com/nebari-dev/nebari/pull/2556
- #2566 Upgrade Jupyterhub ssh image by @arjxn-py in https://github.com/nebari-dev/nebari/pull/2576
- Stop copying unnecessary files into user home directory by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2578
- Include deprecation notes for init/deploy subcommands by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2582
- Only download jar if file doesn't exist by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2588
- Remove unnecessary experimental flag by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2606
- Add typos spell checker to pre-commit by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2568
- Enh 2451 skip conditionals by @BrianCashProf in https://github.com/nebari-dev/nebari/pull/2569
- Improve codespell support: adjust and concentrate config to pyproject.toml and fix more typos by @yarikoptic in https://github.com/nebari-dev/nebari/pull/2583
- Move codespell config to pyproject.toml only by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2611
- Add `depends_on` for bucket encryption by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2615

### New Contributors

- @BrianCashProf made their first contribution in https://github.com/nebari-dev/nebari/pull/2569
- @yarikoptic made their first contribution in https://github.com/nebari-dev/nebari/pull/2583

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.6.1...2024.7.1

## Release 2024.6.1 - June 26, 2024

> NOTE: This release includes an upgrade to the `kube-prometheus-stack` Helm chart, resulting in a newer version of Grafana. When upgrading your Nebari cluster, you will be prompted to have Nebari update some CRDs and delete a DaemonSet on your behalf. If you prefer, you can also run the commands yourself, which will be shown to you. If you have any custom dashboards, you'll also need to back them up by [exporting them as JSON](https://grafana.com/docs/grafana/latest/dashboards/share-dashboards-panels/#export-a-dashboard-as-json), so you can [import them](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/#import-a-dashboard) after upgrading.

### What's Changed

- Fetch JupyterHub roles from Keycloak by @krassowski in https://github.com/nebari-dev/nebari/pull/2447
- Update selector for Start server button to use button tag by @krassowski in https://github.com/nebari-dev/nebari/pull/2464
- Reduce GCP Fixed Costs by 50% by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2453
- Restore JupyterHub updates from PR-2427 by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2465
- Workload identity by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2460
- Fix test using a non-specific selector by @krassowski in https://github.com/nebari-dev/nebari/pull/2475
- add verify=false since we use self signed cert in tests by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2481
- fix forward auth when using custom cert by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2479
- Upgrade to JupyterHub 5.0.0b2 by @krassowski in https://github.com/nebari-dev/nebari/pull/2468
- upgrade instructions for PR 2453 by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2466
- Use Helm Chart for JupyterHub 5.0.0 final by @krassowski in https://github.com/nebari-dev/nebari/pull/2484
- Parse and insert keycloak roles scopes into JupyterHub by @aktech in https://github.com/nebari-dev/nebari/pull/2471
- Add CITATION file by @pavithraes in https://github.com/nebari-dev/nebari/pull/2455
- CI: add azure integration by @fangchenli in https://github.com/nebari-dev/nebari/pull/2061
- Create trivy.yml by @dcmcand in https://github.com/nebari-dev/nebari/pull/2458
- don't run azure deployment on PRs, only on schedule and manual trigger by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2498
- add cloud provider deployment status badges to README.md by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2407
- Upgrade kube-prometheus-stack helm chart by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2472
- upgrade note by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2502
- Remove VSCode from jhub_apps default services by @jbouder in https://github.com/nebari-dev/nebari/pull/2503
- Explicit config by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2294
- fix general node scaling bug for azure by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2517
- Skip running cleanup on pull requests by @aktech in https://github.com/nebari-dev/nebari/pull/2488
- 1792 Add docstrings to `upgrade.py` by @arjxn-py in https://github.com/nebari-dev/nebari/pull/2512
- set's min TLS version for azure storage account to TLS 1.2 by @dcmcand in https://github.com/nebari-dev/nebari/pull/2522
- Fix conda-store and Traefik Grafana Dashboards by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2540
- Implement support for jupyterlab-gallery config by @krassowski in https://github.com/nebari-dev/nebari/pull/2501
- Add option to run CRDs updates and DaemonSet deletion on user's behalf. by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2544

### New Contributors

- @arjxn-py made their first contribution in https://github.com/nebari-dev/nebari/pull/2512

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.5.1...2024.6.1

## Release 2024.5.1 - May 13, 2024

### What's Changed

- make userscheduler run on general node group by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2415
- Upgrade to Pydantic V2 by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2348
- Pydantic2 PR fix by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2421
- remove redundant pydantic class, fix bug by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2426
- Update `python-keycloak` version pins constraints by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2435
- add HERA_TOKEN env var to user pods by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2438
- fix docs link by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2443
- Update allowed admin groups by @aktech in https://github.com/nebari-dev/nebari/pull/2429

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.4.1...2024.5.1

## Release 2024.4.1 - April 20, 2024

### What's Changed

- update azurerm version by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2370
- Get JupyterHub `groups` from Keycloak, support `oauthenticator` 16.3+ by @krassowski in https://github.com/nebari-dev/nebari/pull/2361
- add full names for cloud providers in guided init by @exitflynn in https://github.com/nebari-dev/nebari/pull/2375
- Add middleware to prefix JupyterHub navbar items with /hub. by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2360
- CLN: split #1928, refactor render test by @fangchenli in https://github.com/nebari-dev/nebari/pull/2246
- add trailing slash for jupyterhub proxy paths by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2387
- remove references to deprecated cdsdashboards by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2390
- add default node groups to config by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2398
- Update concurrency settings for Integration tests by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2393
- Make CI/CD Cloud Provider Test Conditional by @tylergraff in https://github.com/nebari-dev/nebari/pull/2369

### New Contributors

- @exitflynn made their first contribution in https://github.com/nebari-dev/nebari/pull/2375

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.3.3...2024.4.1

## Release 2024.3.3 - March 27, 2024

### What's Changed

- get default variable value when following a terraform variable by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2322
- Upgrade Actions versions by @isumitjha in https://github.com/nebari-dev/nebari/pull/2291
- Cleanup spawner logs by @krassowski in https://github.com/nebari-dev/nebari/pull/2328
- Fix loki gateway url when deployed on non-dev namespace by @aktech in https://github.com/nebari-dev/nebari/pull/2327
- Dmcandrew update ruamel.yaml by @dcmcand in https://github.com/nebari-dev/nebari/pull/2315
- upgrade auth0-python version to ultimately resolve CVE-2024-26130 by @tylergraff in https://github.com/nebari-dev/nebari/pull/2314
- remove deprecated code paths by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2349
- Create SECURITY.md by @dcmcand in https://github.com/nebari-dev/nebari/pull/2354
- Set node affinity for more pods to ensure they run on general node pool by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2353
- Deduplicate conda-store in JupyterLab main menu by @krassowski in https://github.com/nebari-dev/nebari/pull/2347
- Pass current namespace to argo via environment variable by @krassowski in https://github.com/nebari-dev/nebari/pull/2317
- PVC for Traefik Ingress (prevent LetsEncrypt throttling) by @kenafoster in https://github.com/nebari-dev/nebari/pull/2352

### New Contributors

- @isumitjha made their first contribution in https://github.com/nebari-dev/nebari/pull/2291
- @tylergraff made their first contribution in https://github.com/nebari-dev/nebari/pull/2314

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.3.2...2024.3.3

## Release 2024.3.2 - March 14, 2024

### What's Changed

- update max k8s versions and remove depreciated api usage in local deploy by @dcmcand in https://github.com/nebari-dev/nebari/pull/2276
- update keycloak image repo by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2312
- Generate random password for Grafana by @aktech in https://github.com/nebari-dev/nebari/pull/2289
- update conda store to 2024.3.1 by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2316
- Switch PyPI release workflow to use trusted publishing by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2323

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.3.1...2024.3.2

## Release 2024.3.1 - March 11, 2024

### What's Changed

- Modify Playwright test to account for changes in JupyterLab UI. by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2232
- Add favicon to jupyterhub theme. by @jbouder in https://github.com/nebari-dev/nebari/pull/2222
- Set min nodes to 0 for worker and user. by @pt247 in https://github.com/nebari-dev/nebari/pull/2168
- Remove `jhub-client` from pyproject.toml by @pavithraes in https://github.com/nebari-dev/nebari/pull/2242
- Include permission validation step to programmatically cloned repos by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2258
- Expose jupyter's preferred dir as a config option by @krassowski in https://github.com/nebari-dev/nebari/pull/2251
- Allow to configure default settings for JupyterLab (`overrides.json`) by @krassowski in https://github.com/nebari-dev/nebari/pull/2249
- Feature/jlab menu customization by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2259
- Add cloud provider to the dask config.json file by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2266
- Fix syntax error in jupyter-server-config Python file by @krassowski in https://github.com/nebari-dev/nebari/pull/2286
- Add "Open VS Code" entry in services by @krassowski in https://github.com/nebari-dev/nebari/pull/2267
- Add Grafana Loki integration by @aktech in https://github.com/nebari-dev/nebari/pull/2156

### New Contributors

- @jbouder made their first contribution in https://github.com/nebari-dev/nebari/pull/2222
- @krassowski made their first contribution in https://github.com/nebari-dev/nebari/pull/2251

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.1.1...2024.3.1

## Release 2024.1.1 - January 17, 2024

### Feature changes and enhancements

- Upgrade conda-store to latest version 2024.1.1
- Add Jhub-Apps
- Add Jupyterlab-pioneer
- Minor improvements and bug fixes

### Breaking Changes

> WARNING: jupyterlab-videochat, retrolab, jupyter-tensorboard, jupyterlab-conda-store and jupyter-nvdashboard are no longer supported in Nebari version and will be uninstalled."

### What's Changed

- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/2176
- Fix logic for dns lookup. by @pt247 in https://github.com/nebari-dev/nebari/pull/2166
- Integrate JupyterHub App Launcher into Nebari by @aktech in https://github.com/nebari-dev/nebari/pull/2185
- Pass in permissions boundary to k8s module by @aktech in https://github.com/nebari-dev/nebari/pull/2153
- Add jupyterlab-pioneer by @aktech in https://github.com/nebari-dev/nebari/pull/2127
- JHub Apps: Filter conda envs by user by @aktech in https://github.com/nebari-dev/nebari/pull/2187
- update upgrade command by @dcmcand in https://github.com/nebari-dev/nebari/pull/2198
- Remove JupyterLab from services list by @aktech in https://github.com/nebari-dev/nebari/pull/2189
- Adding fields to ignore within keycloak_realm by @costrouc in https://github.com/nebari-dev/nebari/pull/2200
- Add Nebari menu item configuration. by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2196
- Disable "Newer update available" popup as default setting by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2192
- Block usage of pip inside jupyterlab by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2191
- Return all environments instead of just those under the user's namespace for jhub-apps by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2206
- Adding a temporary writable directory for conda-store server /home/conda by @costrouc in https://github.com/nebari-dev/nebari/pull/2209
- Add demo repositories mechanism to populate user's space by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2207
- update nebari_workflow_controller and conda_store tags to test rc by @dcmcand in https://github.com/nebari-dev/nebari/pull/2210
- 2023.12.1 release notes by @dcmcand in https://github.com/nebari-dev/nebari/pull/2211
- Make it so that jhub-apps default theme doesn't override by @costrouc in https://github.com/nebari-dev/nebari/pull/2213
- Adding additional theme variables to jupyterhub theme config by @costrouc in https://github.com/nebari-dev/nebari/pull/2215
- updates Current Release to 2024.1.1 by @dcmcand in https://github.com/nebari-dev/nebari/pull/2227

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2023.12.1...2024.1.1

## Release 2023.12.1 - December 15, 2023

### Feature changes and enhancements

- Upgrade conda-store to latest version 2023.10.1
- Minor improvements and bug fixes

### Breaking Changes

> WARNING: Prefect, ClearML and kbatch were removed in this release and upgrading to this version will result in all of them being uninstalled.

### What's Changed

- BUG: fix incorrect config override #2086 by @fangchenli in https://github.com/nebari-dev/nebari/pull/2087
- ENH: add AWS IAM permissions_boundary option #2078 by @fangchenli in https://github.com/nebari-dev/nebari/pull/2082
- CI: cleanup local integration workflow by @fangchenli in https://github.com/nebari-dev/nebari/pull/2079
- ENH: check missing GCP services by @fangchenli in https://github.com/nebari-dev/nebari/pull/2036
- ENH: use packaging for version parsing, add unit tests by @fangchenli in https://github.com/nebari-dev/nebari/pull/2048
- ENH: specify required field when retrieving available gcp regions by @fangchenli in https://github.com/nebari-dev/nebari/pull/2033
- Upgrade conda-store to 2023.10.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/2092
- Add upgrade command for 2023.11.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/2103
- CLN: cleanup typing and typing import in init by @fangchenli in https://github.com/nebari-dev/nebari/pull/2107
- Remove kbatch, prefect and clearml by @iameskild in https://github.com/nebari-dev/nebari/pull/2101
- Fix integration tests, helm-validate script by @iameskild in https://github.com/nebari-dev/nebari/pull/2102
- Re-enable AWS tags support by @iameskild in https://github.com/nebari-dev/nebari/pull/2096
- Update upgrade instructions for 2023.11.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/2112
- Update nebari-git env pins by by @iameskild in https://github.com/nebari-dev/nebari/pull/2113
- Update release notes for 2023.11.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/2114

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2023.11.1...2023.12.1

## Release 2023.11.1 - November 15, 2023

### Feature changes and enhancements

- Upgrade conda-store to latest version 2023 .10.1
- Minor improvements and bug fixes

### Breaking Changes

> WARNING: Prefect, ClearML and kbatch were removed in this release and upgrading to this version will result in all of them being uninstalled.

### What's Changed

- BUG: fix incorrect config override #2086 by @fangchenli in https://github.com/nebari-dev/nebari/pull/2087
- ENH: add AWS IAM permissions_boundary option #2078 by @fangchenli in https://github.com/nebari-dev/nebari/pull/2082
- CI: cleanup local integration workflow by @fangchenli in https://github.com/nebari-dev/nebari/pull/2079
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/2099
- ENH: check missing GCP services by @fangchenli in https://github.com/nebari-dev/nebari/pull/2036
- ENH: use packaging for version parsing, add unit tests by @fangchenli in https://github.com/nebari-dev/nebari/pull/2048
- ENH: specify required field when retrieving available gcp regions by @fangchenli in https://github.com/nebari-dev/nebari/pull/2033
- Upgrade conda-store to 2023.10.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/2092
- Add upgrade command for 2023.11.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/2103
- CLN: cleanup typing and typing import in init by @fangchenli in https://github.com/nebari-dev/nebari/pull/2107
- Remove kbatch, prefect and clearml by @iameskild in https://github.com/nebari-dev/nebari/pull/2101
- Fix integration tests, helm-validate script by @iameskild in https://github.com/nebari-dev/nebari/pull/2102
- Re-enable AWS tags support by @iameskild in https://github.com/nebari-dev/nebari/pull/2096
- Update upgrade instructions for 2023.11.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/2112
- Update nebari-git env pins by by @iameskild in https://github.com/nebari-dev/nebari/pull/2113
- Update release notes for 2023.11.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/2114

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2023.10.1...2023.11.1

## Release 2023.10.1 - October 20, 2023

This release includes a major refactor which introduces a Pluggy-based extension mechanism which allow developers to build new stages. This is the initial implementation
of the extension mechanism and we expect the interface to be refined overtime. If you're interested in developing your own stage plugin, please refer to [our documentation](https://www.nebari.dev/docs/how-tos/nebari-extension-system#developing-an-extension). When you're ready to upgrade, please download this version from either PyPI or Conda-Forge and run the `nebari upgrade -c nebari-config.yaml`
command and follow the instructions

> WARNING: CDS Dashboards was removed in this release and upgrading to this version will result in CDS Dashboards being uninstalled. A replacement dashboarding solution is currently in the works
> and will be integrated soon.

> WARNING: Given the scope of changes in this release, we highly recommend backing up your system before upgrading. Please refer to our [Manual Backup](https://www.nebari.dev/docs/how-tos/manual-backup) documentation for more details.

### Feature changes and enhancements

- Extension Mechanism Implementation in [PR 1833](https://github.com/nebari-dev/nebari/pull/1833)
  - This also includes much stricter schema validation.
- JupyterHub upgraded to 3.1 in [PR 1856](https://github.com/nebari-dev/nebari/pull/1856)'

### Breaking Changes

- While we have tried our best to avoid breaking changes when introducing the extension mechanism, the scope of the changes is too large for us to confidently say there won't be breaking changes.

> WARNING: CDS Dashboards was removed in this release and upgrading to this version will result in CDS Dashboards being uninstalled. A replacement dashboarding solution is currently in the work and will be integrated soon.

> WARNING: We will be removing and ending support for ClearML, Prefect and kbatch in the next release. The kbatch has been functionally replaced by Argo-Jupyter-Scheduler. We have seen little interest in ClearML and Prefect in recent years, and removing makes sense at this point. However if you wish to continue using them with Nebari we encourage you to [write your own Nebari extension](https://www.nebari.dev/docs/how-tos/nebari-extension-system#developing-an-extension).

### What's Changed

- Spinup spot instance for CI with cirun by @aktech in https://github.com/nebari-dev/nebari/pull/1882
- Fix argo-viewer service account reference by @iameskild in https://github.com/nebari-dev/nebari/pull/1881
- Framework for Nebari deployment via pytest for extensive testing by @aktech in https://github.com/nebari-dev/nebari/pull/1867
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/1878
- Test GCP/AWS Deployment with Pytest by @aktech in https://github.com/nebari-dev/nebari/pull/1871
- Bump DigitalOcean provider to latest by @aktech in https://github.com/nebari-dev/nebari/pull/1891
- Ensure path is Path object by @iameskild in https://github.com/nebari-dev/nebari/pull/1888
- enabling viewing hidden files in jupyterlab file explorer by @kalpanachinnappan in https://github.com/nebari-dev/nebari/pull/1893
- Extension Mechanism Implementation by @costrouc in https://github.com/nebari-dev/nebari/pull/1833
- Fix import path in deployment tests & misc by @aktech in https://github.com/nebari-dev/nebari/pull/1908
- pytest:ensure failure on warnings by @costrouc in https://github.com/nebari-dev/nebari/pull/1907
- workaround for mixed string/posixpath error by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1915
- ENH: Remove aws cli, use boto3 by @fangchenli in https://github.com/nebari-dev/nebari/pull/1920
- paginator for boto3 ec2 instance types by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1923
- Update README.md -- fix typo. by @teoliphant in https://github.com/nebari-dev/nebari/pull/1925
- Add more unit tests, add cleanup step for Digital Ocean integration test by @iameskild in https://github.com/nebari-dev/nebari/pull/1910
- Add cleanup step for AWS integration test, ensure disable_prompt is passed through by @iameskild in https://github.com/nebari-dev/nebari/pull/1921
- K8s 1.25 + More Improvements by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1856
- adding lifecycle ignore to eks node group by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1905
- nebari init unit tests by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1931
- Bug fix - JH singleuser environment getting overwritten by @kenafoster in https://github.com/nebari-dev/nebari/pull/1933
- Allow users to specify the Azure RG to deploy into by @iameskild in https://github.com/nebari-dev/nebari/pull/1927
- nebari validate unit tests by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1938
- adding openid connect provider to enable irsa feature by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1903
- nebari upgrade CLI tests by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1963
- CI: Add test coverage by @fangchenli in https://github.com/nebari-dev/nebari/pull/1959
- nebari cli environment variable handling, support, keycloak, dev tests by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1968
- CI: remove empty notebook to fix pre-commit json check by @fangchenli in https://github.com/nebari-dev/nebari/pull/1976
- TYP: fix typing error in plugins by @fangchenli in https://github.com/nebari-dev/nebari/pull/1973
- TYP: fix return class type in hookimpl by @fangchenli in https://github.com/nebari-dev/nebari/pull/1975
- Allow users to specify Azure tags by @iameskild in https://github.com/nebari-dev/nebari/pull/1967
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/1979
- Do not try and add argo envs when disabled by @iameskild in https://github.com/nebari-dev/nebari/pull/1926
- Handle region with care, updates to test suite by @iameskild in https://github.com/nebari-dev/nebari/pull/1930
- remove custom auth from config schema by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1994
- CLI: handle removed dns options in deploy command by @fangchenli in https://github.com/nebari-dev/nebari/pull/1992
- Add API docs by @kcpevey in https://github.com/nebari-dev/nebari/pull/1634
- Upgrade images for jupyterhub-ssh, kbatch by @iameskild in https://github.com/nebari-dev/nebari/pull/1997
- Add permissions to generate_cli_docs workflow by @iameskild in https://github.com/nebari-dev/nebari/pull/2005
- standardize regex and messaging for names by @kenafoster in https://github.com/nebari-dev/nebari/pull/2003
- ENH: specify required fields when retrieving available gcp projects by @fangchenli in https://github.com/nebari-dev/nebari/pull/2008
- Modify JupyterHub networkPolicy to match existing policy by @iameskild in https://github.com/nebari-dev/nebari/pull/1991
- Update package dependencies by @iameskild in https://github.com/nebari-dev/nebari/pull/1986
- CI: Add AWS integration test workflow, clean up by @iameskild in https://github.com/nebari-dev/nebari/pull/1977
- BUG: fix unboundlocalerror in integration test by @fangchenli in https://github.com/nebari-dev/nebari/pull/1999
- Auth0/Github auth-provider config validation fix by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/2009
- terraform upgrade to 1.5.7 by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1998
- cli init repo auto provision fix by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/2012
- Add gcp_cleanup, minor changes by @iameskild in https://github.com/nebari-dev/nebari/pull/2010
- Fix #2024 by @dcmcand in https://github.com/nebari-dev/nebari/pull/2025
- Upgrade conda-store to 2023.9.2 by @iameskild in https://github.com/nebari-dev/nebari/pull/2028
- Add upgrade steps, instructions for 2023.9.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/2029
- CI: add gcp integration test by @fangchenli in https://github.com/nebari-dev/nebari/pull/2049
- CLN: remove flake8 from dependencies by @fangchenli in https://github.com/nebari-dev/nebari/pull/2044
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/2047
- fix typo in guided init for Digital Ocean by @dcmcand in https://github.com/nebari-dev/nebari/pull/2059
- CI: add do integration by @fangchenli in https://github.com/nebari-dev/nebari/pull/2060
- TYP: make all subfolders under kubernetes_services/template non-module by @fangchenli in https://github.com/nebari-dev/nebari/pull/2043
- TYP: fix most typing errors in provider by @fangchenli in https://github.com/nebari-dev/nebari/pull/2038
- Fix link to documentation on Nebari Deployment home page by @aktech in https://github.com/nebari-dev/nebari/pull/2063
- TST: enable timeout config in playwright notebook test by @fangchenli in https://github.com/nebari-dev/nebari/pull/1996
- DEPS: sync supported python version by @fangchenli in https://github.com/nebari-dev/nebari/pull/2065
- Test support for Python 3.12 by @aktech in https://github.com/nebari-dev/nebari/pull/2046
- BUG: fix validation error related to `provider` #2054 by @fangchenli in https://github.com/nebari-dev/nebari/pull/2056
- CI: improve unit test workflow in CI, revert #2046 by @fangchenli in https://github.com/nebari-dev/nebari/pull/2071
- TST: enable exact_match config in playwright notebook test by @fangchenli in https://github.com/nebari-dev/nebari/pull/2027
- CI: move conda build test to separate job by @fangchenli in https://github.com/nebari-dev/nebari/pull/2073
- Revert conda-store to v0.4.14, #2028 by @iameskild in https://github.com/nebari-dev/nebari/pull/2074
- ENH/CI: add mypy config, and CI workflow by @fangchenli in https://github.com/nebari-dev/nebari/pull/2066
- Update upgrade for 2023.10.1 by @kenfoster in https://github.com/nebari-dev/nebari/pull/2080
- Update RELEASE notes, minor fixes by @iameskild in https://github.com/nebari-dev/nebari/pull/2039

### New Contributors

- @kalpanachinnappan made their first contribution in https://github.com/nebari-dev/nebari/pull/1893
- @fangchenli made their first contribution in https://github.com/nebari-dev/nebari/pull/1920
- @teoliphant made their first contribution in https://github.com/nebari-dev/nebari/pull/1925
- @kenafoster made their first contribution in https://github.com/nebari-dev/nebari/pull/1933
- @dcmcand made their first contribution in https://github.com/nebari-dev/nebari/pull/2025

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2023.7.2...2023.10.1

## Release 2023.7.2 - August 3, 2023

This is a hot-fix release that resolves an issue whereby users in the `analyst` group are unable to launch their JupyterLab server because the name of the viewer-specific `ARGO_TOKEN` was mislabeled; see [PR 1881](https://github.com/nebari-dev/nebari/pull/1881) for more details.

### What's Changed

- Fix argo-viewer service account reference by @iameskild in https://github.com/nebari-dev/nebari/pull/1881
- Add release notes for 2023.7.2, update release notes for 2023.7.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/1886

## Release 2023.7.1 - July 21, 2023

> WARNING: CDS Dashboards will be deprecated soon. Nebari `2023.7.1` will be the last release with support for CDS Dashboards integration. A new dashboard sharing mechanism added in the near future, but some releases in the interim will not have dashboard sharing capabilities..

> WARNING: For those running on AWS, upgrading from previous versions to `2023.7.1` requires a [backup](https://www.nebari.dev/docs/how-tos/manual-backup). Due to changes made to the VPC (See [issue 1884](https://github.com/nebari-dev/nebari/issues/1884) for details), Terraform thinks it needs to destroy and reprovision a new VPC which causes the entire cluster to be destroyed and rebuilt.

### Feature changes and enhancements

- Addition of Nebari-Workflow-Controller in [PR 1741](https://github.com/nebari-dev/nebari/pull/1741)
- Addition of Argo-Jupyter-Scheduler in [PR 1832](https://github.com/nebari-dev/nebari/pull/1832)
- Make most of the API private

### Breaking Changes

- As mentioned in the above WARNING, clusters running on AWS should perform a [manual backup](https://www.nebari.dev/docs/how-tos/manual-backup) before running the upgrade to the latest version as changes to the AWS VPC will cause the cluster to be destroyed and redeployed.

### What's Changed

- use conda forge explicitly in conda build test by @pmeier in https://github.com/nebari-dev/nebari/pull/1771
- document that the upgrade command is for all nebari upgrades by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1794
- don't fail CI matrices fast by @pmeier in https://github.com/nebari-dev/nebari/pull/1804
- unvendor keycloak_metrics_spi by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1810
- Dedent fail-fast by @iameskild in https://github.com/nebari-dev/nebari/pull/1815
- support deploying on existing vpc on aws by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1807
- purge most danlging qhub references by @pmeier in https://github.com/nebari-dev/nebari/pull/1802
- Add Argo Workflow Admission controller by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1741
- purge infracost CLI command / CI jobs by @pmeier in https://github.com/nebari-dev/nebari/pull/1820
- remove unused function parameters and CLI flags by @pmeier in https://github.com/nebari-dev/nebari/pull/1725
- purge docs and nox by @pmeier in https://github.com/nebari-dev/nebari/pull/1801
- Add Helm chart lint tool by @viniciusdc in https://github.com/nebari-dev/nebari/pull/1679
- don't set /etc/hosts in CI by @pmeier in https://github.com/nebari-dev/nebari/pull/1729
- remove execute permissions on templates by @pmeier in https://github.com/nebari-dev/nebari/pull/1798
- fix deprecated file deletion by @pmeier in https://github.com/nebari-dev/nebari/pull/1799
- make nebari API private by @pmeier in https://github.com/nebari-dev/nebari/pull/1778
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/1831
- Simplify CI by @iameskild in https://github.com/nebari-dev/nebari/pull/1819
- Fix edge-case where k8s_version is equal to HIGHEST_SUPPORTED_K8S_VER… by @iameskild in https://github.com/nebari-dev/nebari/pull/1842
- add more configuration to enable private clusters on AWS by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1841
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/1851
- AWS gov cloud support by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1857
- Pathlib everywhere by @pmeier in https://github.com/nebari-dev/nebari/pull/1773
- Initial playwright setup by @kcpevey in https://github.com/nebari-dev/nebari/pull/1665
- Changes required for Jupyter-Scheduler integration by @iameskild in https://github.com/nebari-dev/nebari/pull/1832
- Update upgrade command in preparation for release by @iameskild in https://github.com/nebari-dev/nebari/pull/1868
- Add release notes by @iameskild in https://github.com/nebari-dev/nebari/issues/1869

### New Contributors

- @sblair-metrostar made their first contribution in https://github.com/nebari-dev/nebari/pull/1857

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2023.5.1...2023.7.1

### Release 2023.5.1 - May 5, 2023

### Feature changes and enhancements

- Upgrade Argo-Workflows to version 3.4.4

### Breaking Changes

- The Argo-Workflows version upgrade will result in a breaking change if the existing Kubernetes CRDs are not deleted (see the NOTE below for more details).
- There is a minor breaking change for the Nebari CLI version shorthand, previously it `nebari -v` and now to align with Python convention, it will be `nebari -V`.

> NOTE: After installing the Nebari version `2023.5.1`, please run `nebari upgrade -c nebari-config.yaml` to upgrade
> the `nebari-config.yaml`. This command will also prompt you to delete a few Kubernetes resources (specifically
> the Argo-Workflows CRDS and service accounts) before you can upgrade.

### What's Changed

- Use --quiet flag for conda install in CI by @pmeier in https://github.com/nebari-dev/nebari/pull/1699
- improve CLI tests by @pmeier in https://github.com/nebari-dev/nebari/pull/1710
- Fix Existing dashboards by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1723
- Fix dashboards by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1727
- Typo in the conda-store - conda_store key by @costrouc in https://github.com/nebari-dev/nebari/pull/1740
- use -V (upper case) for --version short form by @pmeier in https://github.com/nebari-dev/nebari/pull/1720
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/1692
- improve pytest configuration by @pmeier in https://github.com/nebari-dev/nebari/pull/1700
- fix upgrade command to look for nebari_version instead of qhub_version by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1693
- remove lazy import by @pmeier in https://github.com/nebari-dev/nebari/pull/1721
- fix nebari invocation through python by @pmeier in https://github.com/nebari-dev/nebari/pull/1711
- Update Argo Workflows to latest version by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1639
- Update secret token in release-notes-sync action by @pavithraes in https://github.com/nebari-dev/nebari/pull/1753
- Typo fix in release-notes-sync action by @pavithraes in https://github.com/nebari-dev/nebari/pull/1756
- 🔄 Synced file(s) with nebari-dev/.github by @nebari-sensei in https://github.com/nebari-dev/nebari/pull/1758
- Update path in release-notes-sync action by @pavithraes in https://github.com/nebari-dev/nebari/pull/1757
- Updating heading format in release notes by @pavithraes in https://github.com/nebari-dev/nebari/pull/1761
- Update vault url by @costrouc in https://github.com/nebari-dev/nebari/pull/1752
- Fix? contributor test trigger by @pmeier in https://github.com/nebari-dev/nebari/pull/1734
- Consistent user Experience with y/N. by @AM-O7 in https://github.com/nebari-dev/nebari/pull/1747
- Fix contributor trigger by @pmeier in https://github.com/nebari-dev/nebari/pull/1765
- add more debug output to contributor test trigger by @pmeier in https://github.com/nebari-dev/nebari/pull/1766
- fix copy-paste error by @pmeier in https://github.com/nebari-dev/nebari/pull/1767
- add instructions insufficient permissions of contributor trigger by @pmeier in https://github.com/nebari-dev/nebari/pull/1772
- fix invalid escape sequence by @pmeier in https://github.com/nebari-dev/nebari/pull/1770
- Update AMI in `.cirun.yml` for nebari-dev-ci AWS account by @aktech in https://github.com/nebari-dev/nebari/pull/1776
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/1768
- turn warnings into errors with pytest by @pmeier in https://github.com/nebari-dev/nebari/pull/1774
- purge setup.cfg by @pmeier in https://github.com/nebari-dev/nebari/pull/1781
- improve pre-commit run on GHA by @pmeier in https://github.com/nebari-dev/nebari/pull/1782
- Upgrade to k8s 1.24 by @iameskild in https://github.com/nebari-dev/nebari/pull/1760
- Overloaded dask gateway fix by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1777
- Add option to specify GKE release channel by @iameskild in https://github.com/nebari-dev/nebari/pull/1648
- Update upgrade command, add RELEASE notes by @iameskild in https://github.com/nebari-dev/nebari/pull/1789

### New Contributors

- @pmeier made their first contribution in https://github.com/nebari-dev/nebari/pull/1699
- @AM-O7 made their first contribution in https://github.com/nebari-dev/nebari/pull/1747

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2023.4.1...2023.5.1

## Release 2023.4.1 - April 12, 2023

> NOTE: Nebari requires Kubernetes version 1.23 and Digital Ocean now requires new clusters to run Kubernetes version 1.24. This means that if you are currently running on Digital Ocean, you should be fine but deploying on a new cluster on Digital Ocean is not possible until we upgrade Kubernetes version (see [issue 1622](https://github.com/nebari-dev/nebari/issues/1622) for more details).

### Feature changes and enhancements

- Upgrades and improvements to conda-store including a new user-interface and greater administrator capabilities.
- Idle-culler settings can now be configured directly from the `nebari-config.yaml`.

### What's Changed

- PR: Raise timeout for jupyter session by @ppwadhwa in https://github.com/nebari-dev/nebari/pull/1646
- PR lower dashboard launch timeout by @ppwadhwa in https://github.com/nebari-dev/nebari/pull/1647
- PR: Update dashboard environment by @ppwadhwa in https://github.com/nebari-dev/nebari/pull/1655
- Fix doc link in README.md by @tkoyama010 in https://github.com/nebari-dev/nebari/pull/1660
- PR: Update dask environment by @ppwadhwa in https://github.com/nebari-dev/nebari/pull/1654
- Feature remove jupyterlab news by @costrouc in https://github.com/nebari-dev/nebari/pull/1641
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/1644
- Feat GitHub actions before_script and after_script steps by @costrouc in https://github.com/nebari-dev/nebari/pull/1672
- Remove examples folder by @ppwadhwa in https://github.com/nebari-dev/nebari/pull/1664
- Fix GH action typos by @kcpevey in https://github.com/nebari-dev/nebari/pull/1677
- Github Actions CI needs id-token write permissions by @costrouc in https://github.com/nebari-dev/nebari/pull/1682
- Update AWS force destroy script, include lingering volumes by @iameskild in https://github.com/nebari-dev/nebari/pull/1681
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/1673
- Make idle culler settings configurable from the `nebari-config.yaml` by @iameskild in https://github.com/nebari-dev/nebari/pull/1689
- Update pyproject dependencies and add test to ensure it builds on conda-forge by @iameskild in https://github.com/nebari-dev/nebari/pull/1662
- Retrieve secrets from Vault, fix test-provider CI by @iameskild in https://github.com/nebari-dev/nebari/pull/1676
- Pull PyPI secrets from Vault by @iameskild in https://github.com/nebari-dev/nebari/pull/1696
- Adding newest conda-store 0.4.14 along with superadmin credentials by @costrouc in https://github.com/nebari-dev/nebari/pull/1701
- Update release notes for 2023.4.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/1722

### New Contributors

- @ppwadhwa made their first contribution in https://github.com/nebari-dev/nebari/pull/1646
- @tkoyama010 made their first contribution in https://github.com/nebari-dev/nebari/pull/1660

**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2023.1.1...2023.4.1

## Release 2023.1.1 - January 30, 2023

### What's Changed

- 🔄 Synced file(s) with nebari-dev/.github by @nebari-sensei in https://github.com/nebari-dev/nebari/pull/1588
- Make conda-store file system read-only by default by @alimanfoo in https://github.com/nebari-dev/nebari/pull/1595
- ENH - Switch to ruff and pre-commit.ci by @trallard in https://github.com/nebari-dev/nebari/pull/1602
- Migrate to hatch by @iameskild in https://github.com/nebari-dev/nebari/pull/1545
- Add check_repository_cred function to CLI by @iameskild in https://github.com/nebari-dev/nebari/pull/1605
- Adding jupyterlab-conda-store extension support to Nebari by @costrouc in https://github.com/nebari-dev/nebari/pull/1564
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/1613
- Ensure Argo-Workflow controller containerRuntimeExecutor is set to emissary by @iameskild in https://github.com/nebari-dev/nebari/pull/1614
- Pass `secret_name` to TF scripts when certificate type = existing by @iameskild in https://github.com/nebari-dev/nebari/pull/1621
- Pin Nebari dependencies, set k8s version for GKE by @iameskild in https://github.com/nebari-dev/nebari/pull/1624
- Create aws-force-destroy bash script by @iameskild in https://github.com/nebari-dev/nebari/pull/1611
- Add option for AWS node-groups to run in a single subnet/AZ by @iameskild in https://github.com/nebari-dev/nebari/pull/1428
- Add export-users to keycloak CLI command, add dev CLI command by @iameskild in https://github.com/nebari-dev/nebari/pull/1610
- Unpin packages in default dashboard env by @iameskild in https://github.com/nebari-dev/pull/1628
- Add release notes for 2023.1.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/1629
- Set GKE release_channel to unspecified to prevent auto k8s updates by @iameskild in https://github.com/nebari-dev/nebari/pull/1630
- Update default nebari-dask, nebari image tags by @iameskild in https://github.com/nebari-dev/nebari/pull/1636

### New Contributors

- @pre-commit-ci made their first contribution in https://github.com/nebari-dev/nebari/pull/1613

## Release 2022.11.1 - December 1, 2022

### What's Changed

- cherry-pick Update README logo (#1514) by @aktech in https://github.com/nebari-dev/nebari/pull/1517
- Release/2022.10.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/1527
- Add Note about QHub->Nebari rename in old docs by @pavithraes in https://github.com/nebari-dev/nebari/pull/1543
- 🔄 Synced file(s) with nebari-dev/.github by @nebari-sensei in https://github.com/nebari-dev/nebari/pull/1550
- 🔄 Synced file(s) with nebari-dev/.github by @nebari-sensei in https://github.com/nebari-dev/nebari/pull/1551
- 🔄 Synced file(s) with nebari-dev/.github by @nebari-sensei in https://github.com/nebari-dev/nebari/pull/1555
- 🔄 Synced file(s) with nebari-dev/.github by @nebari-sensei in https://github.com/nebari-dev/nebari/pull/1560
- Small CLI fixes by @iameskild in https://github.com/nebari-dev/nebari/pull/1529
- 🔄 Synced file(s) with nebari-dev/.github by @nebari-sensei in https://github.com/nebari-dev/nebari/pull/1561
- Render github actions configurations as yaml by @aktech in https://github.com/nebari-dev/nebari/pull/1528
- Update "QHub" to "Nebari" in example notebooks by @pavithraes in https://github.com/nebari-dev/nebari/pull/1556
- Update links to Nebari docs in guided init by @pavithraes in https://github.com/nebari-dev/nebari/pull/1557
- CI: Spinup unique cirun runners for each job by @aktech in https://github.com/nebari-dev/nebari/pull/1563
- Issue-1417: Improve Dask workers placement on AWS | fixing a minor typo by @limacarvalho in https://github.com/nebari-dev/nebari/pull/1487
- Update `setup-node` version by @iameskild in https://github.com/nebari-dev/nebari/pull/1570
- Facilitate CI run for contributor PR by @aktech in https://github.com/nebari-dev/nebari/pull/1568
- Action to sync release notes with nebari-docs by @pavithraes in https://github.com/nebari-dev/nebari/pull/1554
- Restore how the dask worker node group is selected by default by @iameskild in https://github.com/nebari-dev/nebari/pull/1577
- Fix skip check for workflows by @aktech in https://github.com/nebari-dev/nebari/pull/1578
- 📝 Update readme by @trallard in https://github.com/nebari-dev/nebari/pull/1579
- MAINT - Miscellaneous maintenance tasks by @trallard in https://github.com/nebari-dev/nebari/pull/1580
- Wait for Test PyPI to upload test release by @iameskild in https://github.com/nebari-dev/nebari/pull/1583
- Add release notes for 2022.11.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/1584

### New Contributors

- @nebari-sensei made their first contribution in https://github.com/nebari-dev/nebari/pull/1550
- @limacarvalho made their first contribution in https://github.com/nebari-dev/nebari/pull/1487

## Release 2022.10.1 - October 28, 2022

### **WARNING**

> The project has recently been renamed from QHub to Nebari. If your deployment is is still managed by `qhub`, performing an inplace upgrade will **IRREVOCABLY BREAK** your deployment. This will cause you to lose any data stored on the platform, including but not limited to, NFS (filesystem) data, conda-store environments, Keycloak users and groups, etc. Please [backup](https://www.nebari.dev/docs/how-tos/manual-backup) your data before attempting an upgrade.

### Feature changes and enhancements

We are happy to announce the first official release of Nebari (formly QHub)! This release lays the groundwork for many exciting new features and improvements to come.

This release introduces several important changes which include:

- a major project name change from QHub to Nebari - [PR 1508](https://github.com/nebari-dev/nebari/pull/1508)
- a switch from the SemVer to CalVer versioning format - [PR 1501](https://github.com/nebari-dev/nebari/pull/1501)
- a new, Typer-based CLI for improved user experience - [PR 1443](https://github.com/Quansight/qhub/pull/1443) + [PR 1519](https://github.com/nebari-dev/nebari/pull/1519)

Although breaking changes are never fun, the Nebari development team believes these changes are important for the immediate and future success of the project. If you experience any issues or have any questions about these changes, feel free to open an [issue on our Github repo](https://github.com/nebari-dev/nebari/issues).

### What's Changed

- Switch to CalVer by @iameskild in https://github.com/nebari-dev/nebari/pull/1501
- Update theme welcome messages to use Nebari by @pavithraes in https://github.com/nebari-dev/nebari/pull/1503
- Name change QHub --> Nebari by @iameskild in https://github.com/nebari-dev/nebari/pull/1508
- qhub/initialize: lazy load attributes that require remote information by @FFY00 in https://github.com/nebari-dev/nebari/pull/1509
- Update README logo reference by @viniciusdc in https://github.com/nebari-dev/nebari/pull/1514
- Add fix, enhancements and pytests for CLI by @iameskild in https://github.com/nebari-dev/nebari/pull/1498
- Remove old CLI + cleanup by @iameskild in https://github.com/nebari-dev/nebari/pull/1519
- Update `skip_remote_state_provision` default value by @viniciusdc in https://github.com/nebari-dev/nebari/pull/1521
- Add release notes for 2022.10.1 in https://github.com/nebari-dev/nebari/pull/1523

### New Contributors

- @pavithraes made their first contribution in https://github.com/nebari-dev/nebari/pull/1503
- @FFY00 made their first contribution in https://github.com/nebari-dev/nebari/pull/1509

**Note: The following releases (v0.4.5 and lower) were made under the name `Quansight/qhub`.**

## Release v0.4.5 - October 14, 2022

Enhancements for this release include:

- Fix reported bug with Azure deployments due to outdated azurerm provider
- All dashboards related conda-store environments are now visible as options for spawning dashboards
- New Nebari entrypoint
- New Typer-based CLI for Qhub (available using new entrypoint)
- Renamed built-in conda-store namespaces and added customization support
- Updated Traefik version to support the latest Kubernetes API

### What's Changed

- Update azurerm version by @tjcrone in https://github.com/Quansight/qhub/pull/1471
- Make CDSDashboards.conda_envs dynamically update from function by @costrouc in https://github.com/Quansight/qhub/pull/1358
- Fix get_latest_repo_tag fn by @iameskild in https://github.com/Quansight/qhub/pull/1485
- Nebari Typer CLI by @asmijafar20 in https://github.com/Quansight/qhub/pull/1443
- Pass AWS `region`, `kubernetes_version` to terraform scripts by @iameskild in https://github.com/Quansight/qhub/pull/1493
- Enable ebs-csi driver on AWS, add region + kubernetes_version vars by @iameskild in https://github.com/Quansight/qhub/pull/1494
- Update traefik version + CRD by @iameskild in https://github.com/Quansight/qhub/pull/1489
- [ENH] Switch default and filesystem name envs by @viniciusdc in https://github.com/Quansight/qhub/pull/1357

### New Contributors

- @tjcrone made their first contribution in https://github.com/Quansight/qhub/pull/1471

### Migration note

If you are upgrading from a version of Nebari prior to `0.4.5`, you will need to manually update your conda-store namespaces
to be compatible with the new Nebari version. This is a one-time migration step that will need to be performed after upgrading to continue using the service. Refer to [How to migrate base conda-store namespaces](https://deploy-preview-178--nebari-docs.netlify.app/troubleshooting#conda-store-compatibility-migration-steps-when-upgrading-to-045) for further instructions.

## Release v0.4.4 - September 22, 2022

### Feature changes and enhancements

Enhancements for this release include:

- Bump `conda-store` version to `v0.4.11` and enable overrides
- Fully decouple the JupyterLab, JupyterHub and Dask-Worker images from the main codebase
  - See https://github.com/nebari-dev/nebari-docker-images for images
- Add support for Python 3.10
- Add support for Terraform binary download for M1 Mac
- Add option to supply additional arguments to ingress from qhub-config.yaml
- Add support for Kubernetes Kind (local)

### What's Changed

- Add support for terraform binary download for M1 by @aktech in https://github.com/Quansight/qhub/pull/1370
- Improvements in the QHub Cost estimate tool by @HarshCasper in https://github.com/Quansight/qhub/pull/1365
- Add Python-3.10 by @HarshCasper in https://github.com/Quansight/qhub/pull/1352
- Add backwards compatibility item to test checklist by @viniciusdc in https://github.com/Quansight/qhub/pull/1381
- add code server version to fix build by @HarshCasper in https://github.com/Quansight/qhub/pull/1383
- Update Cirun.io config to use labels by @aktech in https://github.com/Quansight/qhub/pull/1379
- Decouple docker images by @iameskild in https://github.com/Quansight/qhub/pull/1371
- Set LATEST_SUPPORTED_PYTHON_VERSION as str by @iameskild in https://github.com/Quansight/qhub/pull/1387
- Integrate kind into local deployment to no longer require minikube for development by @costrouc in https://github.com/Quansight/qhub/pull/1171
- Upgrade conda-store to 0.4.7 allow for customization by @costrouc in https://github.com/Quansight/qhub/pull/1385
- [ENH] Bump conda-store to v0.4.9 by @viniciusdc in https://github.com/Quansight/qhub/pull/1392
- [ENH] Add `pyarrow` and `s3fs` by @viniciusdc in https://github.com/Quansight/qhub/pull/1393
- Fixing bug in authentication method in Conda-Store authentication by @costrouc in https://github.com/Quansight/qhub/pull/1396
- CI: Merge test and release to PyPi workflows into one by @HarshCasper in https://github.com/Quansight/qhub/pull/1386
- Update packages in the dashboard env by @iameskild in https://github.com/Quansight/qhub/pull/1402
- BUG: Setting behind proxy setting in conda-store to be aware of http vs. https by @costrouc in https://github.com/Quansight/qhub/pull/1404
- Minor update to release workflow by @iameskild in https://github.com/Quansight/qhub/pull/1406
- Clean up release workflow by @iameskild in https://github.com/Quansight/qhub/pull/1407
- Add release notes for v0.4.4 by @iameskild in https://github.com/Quansight/qhub/pull/1408
- Update Ingress overrides behaviour by @viniciusdc in https://github.com/Quansight/qhub/pull/1420
- Preserve conda-store image permissions by @iameskild in https://github.com/Quansight/qhub/pull/1419
- Add project name to jhub helm chart release name by @iameskild in https://github.com/Quansight/qhub/pull/1422
- Fix for helm extension overrides data type issue by @konkapv in https://github.com/Quansight/qhub/pull/1424
- Add option to disable tls certificate by @iameskild in https://github.com/Quansight/qhub/pull/1421
- Fixing provider=existing for local/existing by @costrouc in https://github.com/Quansight/qhub/pull/1425
- Update release, testing checklist by @iameskild in https://github.com/Quansight/qhub/pull/1397
- Add `--disable-checks` flag to deploy by @iameskild in https://github.com/Quansight/qhub/pull/1429
- Adding option to supply additional arguments to ingress via `ingress.terraform_overrides.additional-arguments` by @costrouc in https://github.com/Quansight/qhub/pull/1431
- Add properties to middleware crd headers by @iameskild in https://github.com/Quansight/qhub/pull/1434
- Restart conda-store worker when new conda env is added to config.yaml by @iameskild in https://github.com/Quansight/qhub/pull/1437
- Pin dask ipywidgets version to `7.7.1` by @viniciusdc in https://github.com/Quansight/qhub/pull/1442
- Set qhub-dask version to 0.4.4 by @iameskild in https://github.com/Quansight/qhub/pull/1470

### New Contributors

- @konkapv made their first contribution in https://github.com/Quansight/qhub/pull/1424

## Release v0.4.3 - July 7, 2022

### Feature changes and enhancements

Enhancements for this release include:

- Integrating Argo Workflow
- Integrating kbatch
- Adding `cost-estimate` CLI subcommand (Infracost)
- Add `panel-serve` as a CDS dashboard option
- Add option to use RetroLab instead of default JupyterLab

### What's Changed

- Update the login/Keycloak docs page by @gabalafou in https://github.com/Quansight/qhub/pull/1289
- Add configuration option so myst parser generates anchors for heading… by @costrouc in https://github.com/Quansight/qhub/pull/1299
- Image scanning by @HarshCasper in https://github.com/Quansight/qhub/pull/1291
- Fix display version behavior by @viniciusdc in https://github.com/Quansight/qhub/pull/1275
- [Docs] Add docs about custom Identity providers for Authentication by @viniciusdc in https://github.com/Quansight/qhub/pull/1273
- Add prefect token var to CI when needed by @viniciusdc in https://github.com/Quansight/qhub/pull/1279
- ci: prevent image scans on main image builds by @HarshCasper in https://github.com/Quansight/qhub/pull/1300
- Integrate `kbatch` by @iameskild in https://github.com/Quansight/qhub/pull/1258
- add `retrolab` to the base jupyter image by @tonyfast in https://github.com/Quansight/qhub/pull/1222
- Update pre-commit, remove vale by @iameskild in https://github.com/Quansight/qhub/pull/1282
- Argo Workflows by @Adam-D-Lewis in https://github.com/Quansight/qhub/pull/1252
- Update minio, postgresql chart repo location by @iameskild in https://github.com/Quansight/qhub/pull/1308
- Fix broken AWS, set minimum desired size to 1, enable 0 scaling by @tylerpotts in https://github.com/Quansight/qhub/pull/1304
- v0.4.2 release notes by @iameskild in https://github.com/Quansight/qhub/pull/1323
- install dask lab ext from main by @iameskild in https://github.com/Quansight/qhub/pull/1321
- Overrides default value for dask-labextension by @viniciusdc in https://github.com/Quansight/qhub/pull/1327
- CI: Add Infracost to GHA CI for infra cost tracking by @HarshCasper in https://github.com/Quansight/qhub/pull/1316
- Add check for highest supported k8s version by @aktech in https://github.com/Quansight/qhub/pull/1336
- Increase the default instance sizes by @peytondmurray in https://github.com/Quansight/qhub/pull/1338
- Add panel-serve as a CDS dashboard option by @iameskild in https://github.com/Quansight/qhub/pull/1070
- Generate QHub Costs via `infracost` by @HarshCasper in https://github.com/Quansight/qhub/pull/1340
- Add release-checklist issue template by @iameskild in https://github.com/Quansight/qhub/pull/1314
- Fix missing import: `rich` : broken qhub init with cloud by @aktech in https://github.com/Quansight/qhub/pull/1353
- Bump qhub-dask version to 0.4.3 by @peytondmurray in https://github.com/Quansight/qhub/pull/1341
- Remove the need for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be set with Digital Ocean deployment by @costrouc in https://github.com/Quansight/qhub/pull/1344
- Revert "Remove the need for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be set with Digital Ocean deployment" by @viniciusdc in https://github.com/Quansight/qhub/pull/1355
- Upgrade kbatch version by @iameskild in https://github.com/Quansight/qhub/pull/1335
- Drop support for python 3.7 in dask environment by @peytondmurray in https://github.com/Quansight/qhub/pull/1354
- Add useful terminal utils to jlab image by @dharhas in https://github.com/Quansight/qhub/pull/1361
- Tweak bashrc by @dharhas in https://github.com/Quansight/qhub/pull/1363
- Fix bug where vscode extensions are not installing by @viniciusdc in https://github.com/Quansight/qhub/pull/1360

### New Contributors

- @gabalafou made their first contribution in https://github.com/Quansight/qhub/pull/1289
- @peytondmurray made their first contribution in https://github.com/Quansight/qhub/pull/1338
- @dharhas made their first contribution in https://github.com/Quansight/qhub/pull/1361

**Full Changelog**: https://github.com/Quansight/qhub/compare/v0.4.1...v0.4.3

## Release v0.4.2 - June 8, 2022

### Incident postmortem

#### Bitnami update breaks post v0.4.0 releases

On June 2, 2022, GitHub user @peytondmurray reported [issue 1306](https://github.com/Quansight/qhub/issues/1306), stating that he was unable to deploy QHub using either the latest release `v0.4.1` or installing `qhub` from `main`. As verified by @peytondmurray and others, during your first `qhub deploy`, the deployment halts and complains about two invalid Helm charts missing from the bitnami `index.yaml`.

[Bitnami's decision to update how long they keep old Helm charts in their index for](https://github.com/bitnami/charts/issues/10539) has essentially broken all post `v0.4.0` versions of QHub.

This is a severe bug that will affect any new user who tries to install and deploy QHub with any version less than `v0.4.2` and greater than or equal to `v0.4.0`.

Given the impact and severity of this bug, the team has decided to quickly cut a hotfix.

#### AWS deployment failing due to old auto-scaler helm chart

On May 27, 2022, GitHub user @tylerpotts reported [issue 1302](https://github.com/Quansight/qhub/issues/1302), stating that he was unable to deploy QHub using the latest release `v0.4.1` (or installing `qhub` from `main`). As described in the original issue, the deployment failed complaining about the deprecated `v1beta` Kubernetes API. This led to the discovery that we were using an outdated `cluster_autoscaler` helm chart.

The solution is to update from `v1beta` to `v1` Kubernetes API for the appropriate resources and update the reference to the `cluster_autoscaler` helm chart.

Given the impact and severity of this bug, the team has decided to quickly cut a hotfix.

### Bug fixes

This release is a hotfix for the issue summarized in the following:

- [issue 1319](https://github.com/Quansight/qhub/issues/1319)
- [issue 1306](https://github.com/Quansight/qhub/issues/1306)
- [issue 1302](https://github.com/Quansight/qhub/issues/1302)

### What's Changed

- Update minio, postgresql chart repo location by @iameskild in [PR 1308](https://github.com/Quansight/qhub/pull/1308)
- Fix broken AWS, set minimum desired size to 1, enable 0 scaling by @tylerpotts in [PR 1304](https://github.com/Quansight/qhub/issues/1304)

## Release v0.4.1 - May 10, 2022

### Feature changes and enhancements

Enhancements for this release include:

- Add support for pinning the IP address of the load balancer via terraform overrides
- Upgrade to Conda-Store to `v0.3.15`
- Add ability to limit JupyterHub profiles based on users/groups

### Bug fixes

This release addresses several bugs with a slight emphasis on stabilizing the core services while also improving the end user experience.

### What's Changed

- [BUG] Adding back feature of limiting profiles for users and groups by @costrouc in [PR 1169](https://github.com/Quansight/qhub/pull/1169)
- DOCS: Add release notes for v0.4.0 release by @HarshCasper in [PR 1170](https://github.com/Quansight/qhub/pull/1170)
- Move ipython config within jupyterlab to docker image with more robust jupyterlab ssh tests by @costrouc in [PR 1143](https://github.com/Quansight/qhub/pull/1143)
- Removing custom dask_gateway from qhub and idle_timeout for dask clusters to 30 min by @costrouc in [PR 1151](https://github.com/Quansight/qhub/pull/1151)
- Overrides.json now managed by qhub configmaps instead of inside docker image by @costrouc in [PR 1173](https://github.com/Quansight/qhub/pull/1173)
- Adding examples to QHub jupyterlab by @costrouc in [PR 1176](https://github.com/Quansight/qhub/pull/1176)
- Bump conda-store version to 0.3.12 by @costrouc in [PR 1179](https://github.com/Quansight/qhub/pull/1179)
- Fixing concurrency not being specified in configuration by @costrouc in [PR 1180](https://github.com/Quansight/qhub/pull/1180)
- Adding ipykernel as default to environment along with ensure conda-store restarted on config change by @costrouc in [PR 1181](https://github.com/Quansight/qhub/pull/1181)
- keycloak dev docs by @danlester in [PR 1184](https://github.com/Quansight/qhub/pull/1184)
- Keycloakdev2 by @danlester in [PR 1185](https://github.com/Quansight/qhub/pull/1185)
- Setting minio storage to by default be same as filesystem size for Conda-Store environments by @costrouc in [PR 1188](https://github.com/Quansight/qhub/pull/1188)
- Bump Conda-Store version in Qhub to 0.3.13 by @costrouc in [PR 1189](https://github.com/Quansight/qhub/pull/1189)
- Upgrade mrparkers to 3.7.0 by @danlester in [PR 1183](https://github.com/Quansight/qhub/pull/1183)
- Mdformat tables by @danlester in [PR 1186](https://github.com/Quansight/qhub/pull/1186)
- [ImgBot] Optimize images by @imgbot in [PR 1187](https://github.com/Quansight/qhub/pull/1187)
- Bump conda-store version to 0.3.14 by @costrouc in [PR 1192](https://github.com/Quansight/qhub/pull/1192)
- Allow terraform init to upgrade providers within version specification by @costrouc in [PR 1194](https://github.com/Quansight/qhub/pull/1194)
- Adding missing **init** files by @costrouc in [PR 1196](https://github.com/Quansight/qhub/pull/1196)
- Release 0.3.15 for Conda-Store by @costrouc in [PR 1205](https://github.com/Quansight/qhub/pull/1205)
- Profilegroups by @danlester in [PR 1203](https://github.com/Quansight/qhub/pull/1203)
- Render `.gitignore`, black py files by @iameskild in [PR 1206](https://github.com/Quansight/qhub/pull/1206)
- Update qhub-dask pinned version by @iameskild in [PR 1224](https://github.com/Quansight/qhub/pull/1224)
- Fix env doc links and add corresponding tests by @aktech in [PR 1216](https://github.com/Quansight/qhub/pull/1216)
- Update conda-store-environment variable `type` by @iameskild in [PR 1213](https://github.com/Quansight/qhub/pull/1213)
- Update release notes - justification for changes in `v0.4.0` by @iameskild in [PR 1178](https://github.com/Quansight/qhub/pull/1178)
- Support for pinning the IP address of the load balancer via terraform overrides by @aktech in [PR 1235](https://github.com/Quansight/qhub/pull/1235)
- Bump moment from 2.29.1 to 2.29.2 in /tests_e2e by @dependabot in [PR 1241](https://github.com/Quansight/qhub/pull/1241)
- Update cdsdashboards to 0.6.1, Voila to 0.3.5 by @danlester in [PR 1240](https://github.com/Quansight/qhub/pull/1240)
- Bump minimist from 1.2.5 to 1.2.6 in /tests_e2e by @dependabot in [PR 1208](https://github.com/Quansight/qhub/pull/1208)
- output check fix by @Adam-D-Lewis in [PR 1244](https://github.com/Quansight/qhub/pull/1244)
- Update panel version to fix jinja2 recent issue by @viniciusdc in [PR 1248](https://github.com/Quansight/qhub/pull/1248)
- Add support for terraform overrides in cloud and VPC deployment for Azure by @aktech in [PR 1253](https://github.com/Quansight/qhub/pull/1253)
- Add test-release workflow by @iameskild in [PR 1245](https://github.com/Quansight/qhub/pull/1245)
- Bump async from 3.2.0 to 3.2.3 in /tests_e2e by @dependabot in [PR 1260](https://github.com/Quansight/qhub/pull/1260)
- [WIP] Add support for VPC deployment for GCP via terraform overrides by @aktech in [PR 1259](https://github.com/Quansight/qhub/pull/1259)
- Update login instructions for training by @iameskild in [PR 1261](https://github.com/Quansight/qhub/pull/1261)
- Add docs for general node upgrade by @iameskild in [PR 1246](https://github.com/Quansight/qhub/pull/1246)
- [ImgBot] Optimize images by @imgbot in [PR 1264](https://github.com/Quansight/qhub/pull/1264)
- Fix project name and domain at None by @pierrotsmnrd in [PR 856](https://github.com/Quansight/qhub/pull/856)
- Adding name convention validator for QHub project name by @viniciusdc in [PR 761](https://github.com/Quansight/qhub/pull/761)
- Minor doc updates by @iameskild in [PR 1268](https://github.com/Quansight/qhub/pull/1268)
- Enable display of Qhub version by @viniciusdc in [PR 1256](https://github.com/Quansight/qhub/pull/1256)
- Fix missing region from AWS provider by @viniciusdc in [PR 1271](https://github.com/Quansight/qhub/pull/1271)
- Re-enable GPU profiles for GCP/AWS by @viniciusdc in [PR 1219](https://github.com/Quansight/qhub/pull/1219)
- Release notes for `v0.4.1` by @iameskild in [PR 1272](https://github.com/Quansight/qhub/pull/1272)

### New Contributors

- @dependabot made their first contribution in [PR 1241](https://github.com/Quansight/qhub/pull/1241)

[**Full Changelog**](https://github.com/Quansight/qhub/compare/v0.4.0...v0.4.1)

## Release v0.4.0.post1 - April 7, 2022

This post-release addresses the a few minor bugs and updates the release notes.
There are no breaking changes or API changes.

- Render `.gitignore`, black py files - [PR 1206](https://github.com/Quansight/qhub/pull/1206)
- Update qhub-dask pinned version - [PR 1224](https://github.com/Quansight/qhub/pull/1224)
- Update conda-store-environment variable `type` - [PR 1213](https://github.com/Quansight/qhub/pull/1213)
- Update release notes - justification for changes in `v0.4.0` - [PR 1178](https://github.com/Quansight/qhub/pull/1178)
- Merge spawner and profile env vars to ensure dashboard sharing vars are provided to dashboard servers - [PR 1237](https://github.com/Quansight/qhub/pull/1237)

## Release v0.4.0 - March 17, 2022

**WARNING**

> If you're looking for a stable version of QHub, please consider `v0.3.14`. The `v0.4.0` has many breaking changes and has rough edges that will be resolved in upcoming point releases.

We are happy to announce the release of `v0.4.0`! This release lays the groundwork for many exciting new features and improvements in the future, stay tuned.

Version `v0.4.0` introduced many design changes along with a handful of user-facing changes that require some justification. Unfortunately as a result of these changes, QHub
instances that are upgraded from previous version to `v0.4.0` will irrevocably break.

Until we have a fully functioning backup mechanism, anyone looking to upgrade is highly encouraged to backup their data, see the
[upgrade docs](https://docs.qhub.dev/en/latest/source/admin_guide/breaking-upgrade.html) and more specifically, the
[backup docs](https://docs.qhub.dev/en/latest/source/admin_guide/backup.html).

These design changes were considered important enough that the development team felt they were warranted. Below we try to highlight a few of the largest changes
and provide justification for them.

- Replace Terraforms resource targeting with staged Terraform deployments.
  - _Justification_: using Terraform resource targeting was never an ideal way of handing off outputs from stage to the next and Terraform explicitly warns its users that it's only
    intended to be used "for exceptional situations such as recovering from errors or mistakes".
- Fully remove `cookiecutter` as a templating mechanism.
  - _Justification_: Although `cookiecutter` has its benefits, we were becoming overly reliant on it as a means of rendering various scripts needed for the deployment. Reading through
    Terraform scripts with scattered `cookiecutter` statements was increasing troublesome and a bit intimidating. Our IDEs are also much happier about this change.
- Removing users and groups from the `qhub-config.yaml` and replacing user management with Keycloak.
  - _Justification_: Up until now, any change to QHub deployment needed to be made in the `qhub-config.yaml` which had the benefit of centralizing any configuration. However it
    also potentially limited the kinds of user management tasks while also causing the `qhub-config.yaml` to balloon in size. Another benefit of removing users and groups from the
    `qhub-config.yaml` that deserves highlighting is that user management no longer requires a full redeployment.

Although breaking changes are never fun, we hope the reasons outlined above are encouraging signs that we are working on building a better, more stable, more flexible product. If you
experience any issues or have any questions about these changes, feel free to open an [issue on our Github repo](https://github.com/Quansight/qhub/issues).

### Breaking changes

Explicit user facing changes:

- Upgrading to `v0.4.0` will require a filesystem backup given the scope and size of the current change set.
  - Running `qhub upgrade` will produce an updated `qhub-config.yaml` and a JSON file of users that can then be imported into Keycloak.
- With the addition of Keycloak, QHub will no longer support `security.authentication.type = custom`.
  - No more users and groups in the `qhub-config.yaml`.

### Feature changes and enhancements

- Authentication is now managed by Keycloak.
- QHub Helm extension mechanism added.
- Allow JupyterHub overrides in the `qhub-config.yaml`.
- `qhub support` CLI option to save Kubernetes logs.
- Updates `conda-store` UI.

### What's Changed

<details>

- Enabling Vale CI with GitHub Actions by @HarshCasper in https://github.com/Quansight/qhub/pull/871
- Qhub upgrade by @danlester in https://github.com/Quansight/qhub/pull/870
- Documentation cleanup by @HarshCasper in https://github.com/Quansight/qhub/pull/873
- \[Docs\] Add Traefik wildcard docs by @viniciusdc in https://github.com/Quansight/qhub/pull/876
- replace deprecated "minikube cache add" with "minikube image load" by @Adam-D-Lewis in https://github.com/Quansight/qhub/pull/880
- Azure Python needs different env var names to Terraform by @danlester in https://github.com/Quansight/qhub/pull/882
- Add notes about broken upgrades by @tylerpotts in https://github.com/Quansight/qhub/pull/877
- Keycloak integration first pass by @danlester in https://github.com/Quansight/qhub/pull/848
- K8s tests - keycloak adduser by @danlester in https://github.com/Quansight/qhub/pull/890
- Documentation cleanup by @HarshCasper in https://github.com/Quansight/qhub/pull/889
- Improvements to templates and readme by @trallard in https://github.com/Quansight/qhub/pull/893
- Keycloak docs by @danlester in https://github.com/Quansight/qhub/pull/901
- DOCS: Add a PR Template by @HarshCasper in https://github.com/Quansight/qhub/pull/900
- Delete existing `.gitlab-ci.yml` when rendering by @iameskild in https://github.com/Quansight/qhub/pull/887
- Qhub Extension (Ready for Review) by @Adam-D-Lewis in https://github.com/Quansight/qhub/pull/886
- Updates to Readme by @trallard in https://github.com/Quansight/qhub/pull/897
- Mirror docker images to ghcr and quay container registry by @aktech in https://github.com/Quansight/qhub/pull/912
- Fix CI: skip failure on cleanup by @aktech in https://github.com/Quansight/qhub/pull/910
- Create and solve envs using mamba by @iameskild in https://github.com/Quansight/qhub/pull/915
- Pin terraform providers by @Adam-D-Lewis in https://github.com/Quansight/qhub/pull/914
- qhub-config.yaml as a secret by @danlester in https://github.com/Quansight/qhub/pull/905
- Setup/Add integration/deployment tests via pytest by @aktech in https://github.com/Quansight/qhub/pull/922
- Disable/Remove the stale bot by @viniciusdc in https://github.com/Quansight/qhub/pull/923
- Integrates Hadolint for Dockerfile linting by @HarshCasper in https://github.com/Quansight/qhub/pull/917
- Reduce minimum nodes in user and dask node pools to 0 for Azure / GCP by @tarundmsharma in https://github.com/Quansight/qhub/pull/723
- Allow jupyterhub.overrides in qhub-config.yaml by @danlester in https://github.com/Quansight/qhub/pull/930
- qhub destroy using targets by @danlester in https://github.com/Quansight/qhub/pull/948
- Take AWS region from AWS_DEFAULT_REGION into qhub-config.yaml on init… by @danlester in https://github.com/Quansight/qhub/pull/950
- cookicutter template out of raw by @danlester in https://github.com/Quansight/qhub/pull/951
- kubernetes-initialization depends_on kubernetes by @danlester in https://github.com/Quansight/qhub/pull/952
- Add timeout to terraform import command by @tylerpotts in https://github.com/Quansight/qhub/pull/949
- Timeout in process (for import) by @danlester in https://github.com/Quansight/qhub/pull/955
- Remove user/groups from YAML by @danlester in https://github.com/Quansight/qhub/pull/956
- qhub upgrade custom auth plus tests by @danlester in https://github.com/Quansight/qhub/pull/946
- Add minimal support `centos` images by @iameskild in https://github.com/Quansight/qhub/pull/943
- Keycloak Export by @danlester in https://github.com/Quansight/qhub/pull/947
- qhub cli tool to save kubernetes logs - `qhub support` by @tarundmsharma in https://github.com/Quansight/qhub/pull/818
- Add docs for deploying QHub to existing EKS cluster by @iameskild in https://github.com/Quansight/qhub/pull/944
- Add jupyterhub-idle-culler to jupyterhub image by @danlester in https://github.com/Quansight/qhub/pull/959
- Robust external container registry by @danlester in https://github.com/Quansight/qhub/pull/945
- use qhub-jupyterhub-theme 0.3.3 to simplify JupyterHub config by @danlester in https://github.com/Quansight/qhub/pull/966
- Get kubernetes version for all cloud providers + pytest refactor by @iameskild in https://github.com/Quansight/qhub/pull/927
- Merge hub.extraEnv env vars by @danlester in https://github.com/Quansight/qhub/pull/968
- DOCS: Removing errors from documentation by @HarshCasper in https://github.com/Quansight/qhub/pull/941
- keycloak.realm_display_name by @danlester in https://github.com/Quansight/qhub/pull/973
- minor updates to keycloak docs by @tylerpotts in https://github.com/Quansight/qhub/pull/977
- CI changes for QHub by @HarshCasper in https://github.com/Quansight/qhub/pull/989
- Update `upgrade` docs and general doc improvements by @iameskild in https://github.com/Quansight/qhub/pull/990
- Remove `scope`, `oauth_callback_url` during upgrade step by @iameskild in https://github.com/Quansight/qhub/pull/997
- Adding Conda-Store to QHub by @costrouc in https://github.com/Quansight/qhub/pull/967
- Fix Jupyterlab docker build by @viniciusdc in https://github.com/Quansight/qhub/pull/1001
- DOCS: Fix broken link in setup doc by @HarshCasper in https://github.com/Quansight/qhub/pull/1006
- Fix Kubernetes local test deployment by @viniciusdc in https://github.com/Quansight/qhub/pull/1002
- Initial commit for auth and stages workflow by @costrouc in https://github.com/Quansight/qhub/pull/1003
- Fix formatting issues with black #1003 by @viniciusdc in https://github.com/Quansight/qhub/pull/1020
- use pyproject.toml and setup.cfg for packaging by @tonyfast in https://github.com/Quansight/qhub/pull/986
- Increase timeout/attempts for keycloak check by @viniciusdc in https://github.com/Quansight/qhub/pull/1023
- Fix issue with traefik issuing certificates with let's encrypt acme by @costrouc in https://github.com/Quansight/qhub/pull/1017
- Fixing cds dashboard conda environments being shown by @costrouc in https://github.com/Quansight/qhub/pull/1025
- Fix input variable support for multiple types by @viniciusdc in https://github.com/Quansight/qhub/pull/1029
- Fix Black/Flake8 problems by @danlester in https://github.com/Quansight/qhub/pull/1039
- Add remote state condition for 01-terraform-state provisioning by @viniciusdc in https://github.com/Quansight/qhub/pull/1042
- Round versions for upgrade and schema by @danlester in https://github.com/Quansight/qhub/pull/1038
- Code Server is now installed via conda, and the Jupyterlab Extension is https://github.com/betatim/vscode-binder/ by @costrouc in https://github.com/Quansight/qhub/pull/1044
- Removing cookiecutter from setup.cfg requirements by @costrouc in https://github.com/Quansight/qhub/pull/1026
- Destroy terraform-state stage when condition match by @viniciusdc in https://github.com/Quansight/qhub/pull/1051
- Fix up adding support for security.keycloak.realm_display_name key by @costrouc in https://github.com/Quansight/qhub/pull/1054
- Move external_container_reg to earlier stage by @danlester in https://github.com/Quansight/qhub/pull/1053
- Adding ability to specify overrides back into keycloak configuration by @costrouc in https://github.com/Quansight/qhub/pull/1055
- Deprecating terraform_modules option since no longer used by @costrouc in https://github.com/Quansight/qhub/pull/1057
- Adding security.shared_users_group option for default users group by @costrouc in https://github.com/Quansight/qhub/pull/1056
- Fix up adding back jupyterhub overrides option by @costrouc in https://github.com/Quansight/qhub/pull/1058
- prevent_deploy flag for safeguarding upgrades by @danlester in https://github.com/Quansight/qhub/pull/1047
- CI: Add layer caching for Docker images by @HarshCasper in https://github.com/Quansight/qhub/pull/1061
- Additions to TCP/DNS stage check, fix 1027 by @iameskild in https://github.com/Quansight/qhub/pull/1063
- FIX: Remove concurrency groups by @HarshCasper in https://github.com/Quansight/qhub/pull/1064
- Stage 08 extensions and realms/logout by @danlester in https://github.com/Quansight/qhub/pull/1069
- Auto create/destroy azure resource group by @viniciusdc in https://github.com/Quansight/qhub/pull/1071
- Add CICD schema and render workflows by @iameskild in https://github.com/Quansight/qhub/pull/1068
- Ensure that the shared folder symlink only exists if user has shared folders by @costrouc in https://github.com/Quansight/qhub/pull/1074
- Adds the ability on render to deleted targeted files or directories by @costrouc in https://github.com/Quansight/qhub/pull/1073
- DOCS: QHub 101 by @HarshCasper in https://github.com/Quansight/qhub/pull/1011
- remove jovyan user by @tylerpotts in https://github.com/Quansight/qhub/pull/1089
- More finely scoped github actions and kubernetes_test build docker images by @costrouc in https://github.com/Quansight/qhub/pull/1088
- Adding clearml overrides by @costrouc in https://github.com/Quansight/qhub/pull/1059
- Reorganizing render, deploy, destroy to unify stages input_vars, tf_objects, checks, and state_imports by @costrouc in https://github.com/Quansight/qhub/pull/1091
- Updates/fixes for rendering CICD workflows by @iameskild in https://github.com/Quansight/qhub/pull/1086
- fix bug in state_01_terraform_state function call by @tylerpotts in https://github.com/Quansight/qhub/pull/1094
- Use paths instead of paths-ignore so that test only run on changes to given paths by @costrouc in https://github.com/Quansight/qhub/pull/1097
- \[ENH\] - Update issue templates by @trallard in https://github.com/Quansight/qhub/pull/1083
- Generate independent objects for terraform-state resources by @viniciusdc in https://github.com/Quansight/qhub/pull/1102
- Complete implementation of destroy which goes through each stage by @costrouc in https://github.com/Quansight/qhub/pull/1103
- Change AWS Kubernetes provider authentication to use data.eks_cluster instead of exec by @costrouc in https://github.com/Quansight/qhub/pull/1107
- Relax qhub destroy to attempt to continue destroying resources by @costrouc in https://github.com/Quansight/qhub/pull/1109
- Breaking upgrade docs (0.4) by @danlester in https://github.com/Quansight/qhub/pull/1087
- Simplify default images by @tylerpotts in https://github.com/Quansight/qhub/pull/1114
- Change group structure by @danlester in https://github.com/Quansight/qhub/pull/1112
- Adding status field to each destroy stage to print status by @costrouc in https://github.com/Quansight/qhub/pull/1116
- Incorrect mapping of values to gcp node group instance types by @costrouc in https://github.com/Quansight/qhub/pull/1117
- FIX: Remove Conda Store from default images by @HarshCasper in https://github.com/Quansight/qhub/pull/1119
- Minor fix to `setup.cfg` by @iameskild in https://github.com/Quansight/qhub/pull/1122
- \[DOC\]- Update contribution guidelines by @trallard in https://github.com/Quansight/qhub/pull/1080
- Adding tests to visit additional endpoints by @costrouc in https://github.com/Quansight/qhub/pull/1118
- Adding tests for juypterhub-ssh, jhub-client, and vs code by @costrouc in https://github.com/Quansight/qhub/pull/1123
- Update Keycloak docs by @iameskild in https://github.com/Quansight/qhub/pull/1093
- Upgrade conda-store v0.3.10 and simplify specification of image by @HarshCasper in https://github.com/Quansight/qhub/pull/1130
- \[ImgBot\] Optimize images by @imgbot in https://github.com/Quansight/qhub/pull/1140
- Adjust Idle culler settings and add internal culling by @viniciusdc in https://github.com/Quansight/qhub/pull/1133
- \[BUG\] Removing jovyan home directory and issue with nss configuration by @costrouc in https://github.com/Quansight/qhub/pull/1142
- \[DOC\] Add `troubleshooting` docs by @iameskild in https://github.com/Quansight/qhub/pull/1139
- Update user login guides by @viniciusdc in https://github.com/Quansight/qhub/pull/1144
- \[ImgBot\] Optimize images by @imgbot in https://github.com/Quansight/qhub/pull/1146
- Workaround for kubernetes-client version issue by @iameskild in https://github.com/Quansight/qhub/pull/1148
- Make the commit of the terraform rendering optional (replaces PR 995) by @iameskild in https://github.com/Quansight/qhub/pull/1149
- Fix typos in user guide docs by @ericdatakelly in https://github.com/Quansight/qhub/pull/1154
- Minor docs clean up for v0.4.0 release by @iameskild in https://github.com/Quansight/qhub/pull/1155
- Read-the-docs and documentation updates by @tonyfast in https://github.com/Quansight/qhub/pull/1153
- Add markdown formatter for doc wrapping by @viniciusdc in https://github.com/Quansight/qhub/pull/1152
- remove deprecated param `count` from `.cirun.yml` by @aktech in https://github.com/Quansight/qhub/pull/1164
- Use qhub-bot for keycloak deployment/check by @iameskild in https://github.com/Quansight/qhub/pull/1167
- Only list active conda-envs for dask-gateway by @iameskild in https://github.com/Quansight/qhub/pull/1162

</details>

### New Contributors

- @imgbot made their first contribution in https://github.com/Quansight/qhub/pull/1140
- @ericdatakelly made their first contribution in https://github.com/Quansight/qhub/pull/1154

**Full Changelog**: https://github.com/Quansight/qhub/compare/v0.3.13...v0.4.0

## Release 0.3.13 - October 13, 2021

### Breaking changes

- No known breaking changes

### Feature changes and enhancements

- Allow users to specify external Container Registry ([#741](https://github.com/Quansight/qhub/pull/741))
- Integrate Prometheus and Grafana into QHub ([#733](https://github.com/Quansight/qhub/pull/733))
- Add Traefik Dashboard ([#797](https://github.com/Quansight/qhub/pull/797))
- Make ForwardAuth optional for ClearML ([#830](https://github.com/Quansight/qhub/pull/830))
- Include override configuration for Prefect Agent ([#813](https://github.com/Quansight/qhub/pull/813))
- Improve authentication type checking ([#834](https://github.com/Quansight/qhub/pull/834))
- Switch to pydata Sphinx theme ([#805](https://github.com/Quansight/qhub/pull/805))

### Bug fixes

- Add force-destroy command (only for AWS at the moment) ([#694](https://github.com/Quansight/qhub/pull/694))
- Include namespace in conda-store PVC ([#716](https://github.com/Quansight/qhub/pull/716))
- Secure ClearML behind ForwardAuth ([#721](https://github.com/Quansight/qhub/pull/721))
- Fix connectivity issues with AWS EKS via Terraform ([#734](https://github.com/Quansight/qhub/pull/734))
- Fix conda-store pod eviction and volume conflicts ([#740](https://github.com/Quansight/qhub/pull/740))
- Update `remove_existing_renders` to only delete QHub related files/directories ([#800](https://github.com/Quansight/qhub/pull/800))
- Reduce number of AWS subnets down to 4 to increase the number of available nodes by a factor of 4 ([#839](https://github.com/Quansight/qhub/pull/839))

## Release 0.3.11 - May 7, 2021

### Breaking changes

### Feature changes and enhancements

- better validation messages on github auto provisioning

### Bug fixes

- removing default values from pydantic schema which caused invalid yaml files to unexpectedly pass validation
- make kubespawner_override.environment overridable (prior changes were overwritten)

## Release 0.3.10 - May 6, 2021

### Breaking changes

- reverting `qhub_user` default name to `jovyan`

### Feature changes and enhancements

### Bug fixes

## Release 0.3.9 - May 5, 2021

### Breaking changes

### Feature changes and enhancements

### Bug fixes

- terraform formatting in cookiecutter for enabling GPUs on GCP

## Release 0.3.8 - May 5, 2021

### Breaking changes

### Feature changes and enhancements

- creating releases for QHub simplified
- added an image for overriding the dask-gateway being used

### Bug fixes

- dask-gateway exposed by default now properly
- typo in cookiecutter for enabling GPUs on GCP

## Release 0.3.7 - April 30, 2021

### Breaking changes

### Feature changes and enhancements

- setting `/bin/bash` as the default terminal

### Bug fixes

- `jhsingle-native-proxy` added to the base jupyterlab image

## Release 0.3.6 - April 29, 2021

### Breaking changes

- simplified bash jupyterlab image to no longer have dashboard packages panel, etc.

### Feature changes and enhancements

- added emacs and vim as default editors in image
- added jupyterlab-git and jupyterlab-sidecar since they now support 3.0
- improvements with `qhub destroy` cleanly deleting resources
- allow user to select conda environments for dashboards
- added command line argument `--skip-terraform-state-provision` to allow for skipping terraform state provisioning in `qhub deploy` step
- no longer render `qhub init` `qhub-config.yaml` file in alphabetical order
- allow user to select instance sizes for dashboards

### Bug fixes

- fixed gitlab-ci before_script and after_script
- fixed jovyan -> qhub_user home directory path issue with dashboards

## Release 0.3.5 - April 28, 2021

### Breaking changes

### Feature changes and enhancements

- added a `--skip-remote-state-provision` flag to allow `qhub deploy` within CI to skip the remote state creation
- added saner defaults for instance sizes and jupyterlab/dask profiles
- `qhub init` no longer renders `qhub-config.yaml` in alphabetical order
- `spawn_default_options` to False to force dashboard owner to pick profile
- adding `before_script` and `after_script` key to `ci_cd` to allow customization of CI process

### Bug fixes

## Release 0.3.4 - April 27, 2021

### Breaking changes

### Feature changes and enhancements

### Bug fixes

- remaining issues with ci_cd branch not being fully changed

## Release 0.3.3 - April 27, 2021

### Breaking changes

### Feature changes and enhancements

### Bug fixes

- Moved to ruamel as yaml parser to throw errors on duplicate keys
- fixed a url link error in cds dashboards
- Azure fixes to enable multiple deployments under one account
- Terraform formatting issue in acme_server deployment
- Terraform errors are caught by qhub and return error code

### Breaking changes

## Release 0.3.2 - April 20, 2021

### Bug fixes

- prevent gitlab-ci from freezing on gitlab deployment
- not all branches were configured via the `branch` option in `ci_cd`

## Release 0.3.1 - April 20, 2021

### Feature changes an enhancements

- added gitlab support for CI
- `ci_cd` field is now optional
- AWS provider now respects the region set
- More robust errors messages in cli around project name and namespace
- `git init` default branch is now `main`
- branch for CI/CD is now configurable

### Bug fixes

- typo in `authenticator_class` for custom authentication

## Release 0.3.0 - April 14, 2021

### Feature changes and enhancements

- Support for self-signed certificate/secret keys via kubernetes secrets
- [jupyterhub-ssh](https://github.com/yuvipanda/jupyterhub-ssh) (`ssh` and `sftp` integration) accessible on port `8022` and `8023` respectively
- VSCode([code-server](https://github.com/cdr/code-server)) now provided in default image and integrated with jupyterlab
- [Dask Gateway](https://gateway.dask.org/) now accessible outside of cluster
- Moving fully towards traefik as a load balancer with tight integration with dask-gateway
- Adding ability to specify node selector label for general, user, and worker
- Ability to specify `kube_context` for local deployments otherwise will use default
- Strict schema validation for `qhub-config.yaml`
- Terraform binary is auto-installed and version managed by qhub
- Deploy stage will auto render by default removing the need for render command for end users
- Support for namespaces with qhub deployments on kubernetes clusters
- Full JupyterHub theming including colors now.
- JupyterHub docker image now independent from zero-to-jupyterhub.
- JupyterLab 3 now default user Docker image.
- Implemented the option to locally deploy QHub allowing for local testing.
- Removed the requirement for DNS, authorization is now password-based (no more OAuth requirements).
- Added option for password-based authentication
- CI now tests local deployment on each commit/PR.
- QHub Terraform modules are now pinned to specific git branch via `terraform_modules.repository` and `terraform_modules.ref`.
- Adds support for Azure cloud provider.

### Bug fixes

### Breaking changes

- Terraform version is now pinned to specific version
- `domain` attributed in `qhub-config.yaml` is now the url for the cluster

### Migration guide

0. Version `<version>` is in format `X.Y.Z`
1. Create release branch `release-<version>` based off `main`
2. Ensure full functionality of QHub this involves at a minimum ensuring

- \[ \] GCP, AWS, DO, and local deployment
- \[ \] "Let's Encrypt" successfully provisioned
- \[ \] Dask Gateway functions properly on each
- \[ \] JupyterLab functions properly on each

3. Increment the version number in `qhub/VERSION` in format `X.Y.Z`
4. Ensure that the version number in `qhub/VERSION` is used in pinning QHub in the github actions `qhub/template/{{ cookiecutter.repo_directory }}/.github/workflows/qhub-ops.yaml`
   in format `X.Y.Z`
5. Create a git tag pointing to the release branch once fully tested and version numbers are incremented `v<version>`

---

## Release 0.2.3 - February 5, 2021

### Feature changes, and enhancements

- Added conda prerequisites for GUI packages.
- Added `qhub destroy` functionality that tears down the QHub deployment.
- Changed the default repository branch from `master` to `main`.
- Added error message when Terraform parsing fails.
- Added templates for GitHub issues.

### Bug fixes

- `qhub deploy -c qhub-config.yaml` no longer prompts unsupported argument for `load_config_file`.
- Minor changes on the Step-by-Step walkthrough on the docs.
- Revamp of README.md to make it concise and highlight Nebari Slurm.

### Breaking changes

- Removed the registry for DigitalOcean.

## Thank you for your contributions!

> [Brian Larsen](https://github.com/brl0), [Rajat Goyal](https://github.com/RajatGoyal), [Prasun Anand](https://github.com/prasunanand), and
> [Rich Signell](https://github.com/rsignell-usgs) and [Josef Kellndorfer](https://github.com/jkellndorfer) for the insightful discussions.



---
File: /nebari-docs/docs/docs/tutorials/argo-workflows-walkthrough.md
---

---
id: argo-workflows-walkthrough
title: Argo Workflows Walkthrough
description: A walk through several example workflows
---

# Argo Workflows Walkthrough

## Introduction

Using a workflow manager can help you automate ETL pipelines, schedule regular
analysis, or just chain together a sequence of functions. Argo is available on
Nebari for workflow management. If you haven't already, check out the
[introductory documentation on using Argo](/how-tos/using-argo.md).

For this tutorial we'll be using the
[Hera](https://hera-workflows.readthedocs.io/) interface to Argo. This will
allow us to write a workflow script in Python.

## The most basic of all examples

The following workflow is perhaps the simplest possible workflow. There are
quite a few bits here so we'll step through it.

### Global Configuration

Nebari sets up several environment variables with tokens, etc. that enable us to
use Argo more smoothly. However, there are two global configuration settings
that we'll need to manually add to each workflow script.

```python
from hera.shared import global_config
import os

global_config_host = f"https://{os.environ['ARGO_SERVER'].rsplit(':')[0]}{os.environ['ARGO_BASE_HREF']}/"
global_config.host = global_config_host
global_config.token = os.environ['HERA_TOKEN']
```

### Workflow Labels

Next, we'll set some labels on our workflows. Because Nebari uses a service
account token by default, we need to tell Argo which user we are. We also need
to tell Argo to use the Nebari Workflow Controller so that we have access to
our Nebari file system and conda environments from within the Argo pod
([more information](/how-tos/using-argo.md#access-your-nebari-environments-and-file-system-while-on-an-argo-pod-beta)).

The workflow labels must be hexadecimal ASCII while the usernames have no such
constraint so we have a helper function `sanitize_labels` to ensure that our
label is valid for Argo.

```python
import re

def sanitize_label(label: str) -> str:
    """
    The sanitize_label function converts all characters that are not
    alphanumeric or a - to their hexadecimal ASCII equivalent. This is
    because kubernetes will complain if certain characters are being
    used. This is the same approach taken by Jupyter for sanitizing.

    On the Nebari Workflow Controller, there is a `desanitize_label`
    function that reverses these changes so we can then perform a user
    look up.

    >>> sanitize_label("user@email.com")
    user-40email-2ecom

    Parameters
    ----------
    label: str
        Username to be sanitized (typically Jupyterhub username)

    Returns
    -------
    Hexadecimall ASCII equivalent of `label`

    """
    label = label.lower()
    pattern = r"[^A-Za-z0-9]"
    return re.sub(pattern, lambda x: "-" + hex(ord(x.group()))[2:], label)



username = os.environ["JUPYTERHUB_USER"]
labels = {
    "workflows.argoproj.io/creator-preferred-username": sanitize_label(username),
    'jupyterflow-override': 'true',
}
```

### Time to Live Strategy

By default, Argo does not destroy servers after completion of a workflow.
Because this can cause substantial unexpected cloud costs, we _highly_
recommend _always_ setting the "Time to live strategy", or `TTLStrategy` on
every workflow.

:::note
The Argo UI will only show the workflow details until the
`TTLStrategy` time has elapsed so make sure you have enough time to evaluate
logs, etc. before those details are removed.
:::

```python
from hera.workflows.models import TTLStrategy

DEFAULT_TTL = 90
ttl_strategy = TTLStrategy(
        seconds_after_completion=DEFAULT_TTL,
        seconds_after_success=DEFAULT_TTL,
        seconds_after_failure=DEFAULT_TTL,
    )
```

### Extra parameters

We also will be setting a `node_selector` parameter, but it is optional. If you
do not include it, Argo will run on your current user instance. If you include
it, Argo will run on the server type that you request. These server types
correspond to the names in your `nebari-config.yml`. Nebari Jupyter instances
are always in the `user` group so that's a good place to start, but you may
want to use other CPU or GPU configurations that have been specified in your
config. Its also important to note that if the node pool is limited to (1) node
in the config, Argo will not be able to spin up. Also note that key in this
dictionary refers to the cloud-specific kubernetes node selector label. For
example, AWS uses "eks.amazonaws.com/nodegroup" while GCP uses
"cloud.google.com/gke-nodepool".

The `namespace` parameter is set to `dev` by default, but Nebari sets it up
as part of the environment variables so we'll pull it from there. The
`generate_name` parameter allows us to give our job a prefix and Argo will add
a suffix to ensure uniqueness. Lastly, we'll give the workflow an `entrypoint`.
This parameter needs to match with the name of the `Steps` parameter you're
using (or `DAG`).

### Workflow constructor

Let's put this all together and have a closer look.

```python
from hera.workflows import Steps, script
from hera.workflows import Workflow

@script()
def echo(message: str):
    print(message)


with Workflow(
    generate_name="hello-user",
    entrypoint="steps",
    node_selector={"eks.amazonaws.com/nodegroup": "user"},
    labels=labels,
    namespace=os.environ["ARGO_NAMESPACE"],
    ttl_strategy=ttl_strategy,
) as w:
    with Steps(name="steps"):
        echo(arguments={"message": "hello"})

w.create()
```

Workflows are essentially managing a series of tasks for us. There are two basic
mechanisms to construct these in Argo - `Steps` and `DAG`s.

For this example, we've used `Steps`. When you build your workflow with `Steps`
as a context manager as we've done here, you can add as many methods as you'd
like (for example, duplicating the call to `echo()`) and Argo will separate
out these commands into individual DAG points in a series.

For example, if I wanted to run two separate functions, it might look like this

```python
...
    with Steps(name="steps"):
        echo(arguments={"message": "hello"})
        echo(arguments={"message": "goodbye"})
```

Each step would have its own resource management and logs within Argo. You can
also tell Hera that the function calls within the `Steps` context manager
should be run in parallel.

If you'd like even more control over your workflow, for example diamond
or branching workflows, the `DAG` constructor will allow you to specify that
level of complexity.

## Beyond "Hello World"

We've proven that we can run **something** on Argo, but what about actually
running some Python code? Let's review the requirements.

We've already discussed setting the `jupyter-overrides` label which tells
Nebari to mount our home directory and the conda environments onto our Argo
pod. We will also need to use a Docker image which has conda set up and
initialized. We'll grab the Nebari Jupyter image. This has the added benefit
of bringing parity between running a code on your Nebari instance and running
on Argo.

### "Argo Assistant" code

As you've seen, we're creating quite a bit of peripheral code and we're about
to add even more. Let's bring some structure in to help us organize things.
For this, we have a little "Argo Assistant" code that will help us out.

```python
import logging
import os
import subprocess
from pathlib import Path

from hera.workflows import Container, Parameter, Steps, Workflow
from hera.workflows.models import TTLStrategy

LOGGER = logging.getLogger()

DEFAULT_TTL = 90  # seconds
DEFAULT_ARGO_NODE_TYPE = "user"
DEFAULT_K8S_SELECTOR_LABEL = "eks.amazonaws.com/nodegroup"


class NebariWorkflow(Workflow):
    """Hera Workflow object with required/reasonable default for running
    on Nebari
    """

    def __init__(self, **kwargs):
        super().__init__(**kwargs)

        if "ttl_strategy" in kwargs.keys():
            self.ttl_strategy = kwargs["ttl_strategy"]
        else:
            self.ttl_strategy = TTLStrategy(
                seconds_after_completion=DEFAULT_TTL,
                seconds_after_success=DEFAULT_TTL,
                seconds_after_failure=DEFAULT_TTL,
            )

        if "node_selector" in kwargs.keys():
            self.node_selector = kwargs["node_selector"]
        else:
            self.node_selector = {DEFAULT_K8S_SELECTOR_LABEL: DEFAULT_ARGO_NODE_TYPE}

        self.namespace = os.environ["ARGO_NAMESPACE"]

        self.labels = {
            "workflows.argoproj.io/creator-preferred-username": sanitize_label(username),
            'jupyterflow-override': 'true',
        }


def create_conda_command(
    script_path,
    conda_env,
    stdout_path="stdout.txt",
):
    """Workflows need to be submitted via a bash command that runs a
    python script. This function creates a conda run command that
    will run a script from a given location using a given conda
    environment.

    Parameters
    ----------
    script_path: str
        Path to the python script (including extension) to be run on Argo
    conda_env: str
        Conda environment name in which to the run the `script_path`
    stdout_path: str
        Local Nebari path (for your user) for standard output from
        the given script. Defaults to `stdout.txt`.

    Returns
    -------
    String bash command
    """

    conda_command = f'conda run -n {conda_env} python "{script_path}" >> {stdout_path}'
    return conda_command


def create_bash_container(name="bash-container"):
    """Create a workflow container that is able to receive bash commands"""
    bash_container = Container(
        name="bash-container",
        image="thiswilloverridden",
        inputs=[
            Parameter(name="bash_command")
        ],  # inform argo that an input called bash_command is coming
        command=["bash", "-c"],
        args=["{{inputs.parameters.bash_command}}"],  # use the input parameter
    )
    return bash_container


def submit_argo_script(script_path, conda_env, stdout_path="stdout.txt"):
    """Submit a script to be run via Argo in a specific environment"""
    validated = validate_submission(script_path, conda_env)

    if not validated:
        raise RuntimeError("Unable to submit Argo workflow")

    conda_command = create_conda_command(script_path, conda_env, stdout_path)

    LOGGER.debug("Submitting command {conda_command} to Argo")

    with NebariWorkflow(
        generate_name="workflow-name-",
        entrypoint="steps",
    ) as w:
        bash_container = create_bash_container()
        with Steps(
            name="steps",  # must match Workflow entrypoint
            annotations={"go": "here"},
        ):
            bash_container(
                name="step-name",
                arguments=[Parameter(name="bash_command", value=conda_command)],
            )

    workflow = w.create()
    return workflow

```

Next, you'll need to create a python script and a conda environment. Then to
submit the workflow to Argo you would run the high level command:

```
path = '/path/to/pyfile.py'
nebari_conda_env = 'analyst-workflow-env'
submit_argo_script(path, nebari_conda_env)
```

Now you can go to the Argo UI and monitor progress!

## Conclusion

Well done! You've learned how to submit a python workflow to Argo and have a
few extra tools to help you along.



---
File: /nebari-docs/docs/docs/tutorials/create-dashboard.md
---

---
id: create-dashboard
title: Create, deploy, and share dashboards and apps
description: Quickly build and deploy a panel dashboard in Nebari
---

# Create, deploy, and share dashboards and apps

Analyzing data with visualizations provides insights, and a dashboard stitches these insights into a meaningful story. There are many great open source dashboarding tools out there that you can use to organize and display your data in an engaging and digestible way.

In this tutorial, you'll learn how to create a new dashboard with [Panel](https://panel.holoviz.org/) within Nebari. You'll also learn how to share your newly created dashboard with other users using JHub App Launcher.

:::important
JHub App Launcher was added in Nebari version `2024.1.1`.
Until version `2023.7.1`, Nebari used [CDS Dashboards](https://cdsdashboards.readthedocs.io/en/stable/) for dashboard sharing.
This page has instructions for both tools.
Since, CDS Dashboards is deprecated, the documentation will be removed soon.
:::

## Supported frameworks

This tutorial demonstrates a Panel dashboard built with HoloViews and Bokeh as the backend, but Nebari supports several other frameworks:

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<Tabs>
<TabItem value="jhub-apps" label="JHub App Launcher" default>

JHub App launcher supports `Panel`, `Bokeh`, `Streamlit`, `Plotly Dash`, `Voila`, `Gradio`, `JupyterLab`, and `Any generic Python command`.

In order to use JHub App to create apps, ensure it is enabled in your Nebari deployment. See the [JHub App Launcher docs][jub-app-launcher] for instructions on enabling and using JHub App in Nebari.

</TabItem>
<TabItem value="cds-dashboards" label="CDS Dashboards">

CDS Dashboards supports `Panel`, `Bokeh`, `Voila`, `Streamlit`, and `Plotly`.

</TabItem>
</Tabs>

## Create the dashboard

### 1. Create environment and notebook

[Create a new environment in conda-store][create-env] for your work with the libraries needed to run your notebook.

<Tabs>
<TabItem value="jhub-apps" label="JHub App Launcher" default>

To use JHub Apps, your environment must include `jhub-apps` and the corresponding dashboard/app creation framework, in addition to other libraries required in the notebook.

Hence, for this tutorial:

```yaml
- pandas
- panel
- holoviews
- bokeh
- jupyter_bokeh
- jhub-apps
```

</TabItem>
<TabItem value="cds-dashboards" label="CDS Dashboards">

When deploying an app via CDS Dashboards, you need to have `cdsdashboards-singleuser` installed in your environment. This allows your environment to show up on the environment options menu when creating your app.

Furthermore, with each framework, you will need to make sure that the specific framework you are deploying is installed. You will need to install `streamlit` for a streamlit app, `panel` for a panel app, etc.

Finally, when creating your app, make note of the packages you are using to run the app locally. These will also need to be added to your environment you will be using to create the dashboard.

For this tutorial:

```yaml
- pandas
- panel
- holoviews
- bokeh
- jupyter_bokeh
- cdsdashboards-singleuser>=0.6.2
```

</TabItem>
</Tabs>

[Launch JupyterLab in Nebari][login-keycloak], create a new Jupyter Notebook with a meaningful name (such as `panel-trees-dashboard.ipynb`), and select the environment your created for this notebook from the `select kernel` dropdown (this dropdown menu is located in the top right corner of your notebook).

### 2. Create a panel dashboard

Copy the code below into a `code` cell of your notebook:

```python title="panel-trees-dashboard.ipynb"
import pandas as pd
import holoviews as hv
from bokeh.models import HoverTool
import panel as on

hv.extension('bokeh')
on.extension()

# creating a sample dataset
data_trees = { 'species_name': ['live oak', 'pecan', 'bur oak', 'cedar elm'],
              'avg_diameter_inch': [20, 30, 40, 35]
            }

df = pd.DataFrame(data_trees)

# adding curve/line and bar plots
plot_bar = hv.Bars(df, 'species_name', 'avg_diameter_inch')
plot_curve = hv.Curve(df)

# creating hover tooltip
hover = HoverTool(tooltips=[("avg diameter", "@avg_diameter_inch"),
                            ("species", "@species_name")])
# plot customization
combine_plot = plot_bar.opts(tools=[hover]) + plot_curve.opts(line_dash='dashed')

# creating a dashboard using panel
on.Row(combine_plot).servable()
```

You can run all the cells in your notebook and view the Panel dashboard using the "Preview with Panel" button in the notebook toolbar:

![`About 🌳 - Species and more` dashboard screenshot displaying a bar and line chart of avg_diameter_inch vs species_name](/img/tutorials/trees-dashboard-example.png)

This interactive feature of Panel makes it possible to rapidly prototype and iterate on dashboards.
Feel free to add more plots or different styles to your plots!

## Deploy the dashboard

<Tabs>
<TabItem value="jhub-apps" label="JHub App Launcher" default>

1. On the Nebari Home Page (from JupyterLab, click on the Nebari logo in the top right corner or go to `File` -> `Home`) click on **"Create App"** to create a new web application for your dashboard.
2. Follow the [general instructions](https://jhub-apps.nebari.dev/docs/create-apps/general-app) from the JHub Apps documentation to fill out the `Create app` form.
3. Click **Next**. You'll be redirected to the Spawner profile page. This page will allow you to select the server in which you want your app to run. These options will vary based on the setup of your Nebari deployment (which server types are available overall) and the permissions of your user (which server types you personally have access to).
4. JHub App Launcher will deploy your app (which can take several minutes to complete) and automatically redirect you to it.

Your dashboard app will be available in the Nebari Home page, under "My Apps". If you allowed shared access, it will be available under "Shared Apps" for those with whom you have shared the app.

</TabItem>

<TabItem value="cds-dashboards" label="CDS Dashboard" default>

In this section, you'll use CDS Dashboards ((Nebari v2023.7.1 or earlier)) to publish and share your newly created `panel` dashboard.

:::warning
CDS Dashboards has been deprecated in 2023.9.1. Nebari 2023.7.1 is the last release that support CDS Dashboards.
:::

To begin, click on the top left tab navigate to `File` -> `Hub Control Panel` -> `Dashboards`.

![JupyterLab expanded File menu - Hub Control Panel is highlighted with a surrounding purple box](/img/tutorials/nebari_jupyterlab_file_menu.png)

Click on the button `New Dashboard`. You will now be presented with a new window where you'll need to provide additional details for your dashboard (see image below for reference).

![CDS dashboard configuration screenshot](/img/tutorials/window_dashboard_configuration_example.png)

1. Give your dashboard a name, for example, `Trees`. This name will be the name of your shareable dashboard, so make sure to give this a meaningful name.
2. Add a short description, for example, `Insights and more`.
3. Set the correct user-access permission (optional). This setting allows you to share your dashboard with all the other users on your Nebari deployment or select specific users.
4. Select the code source for your panel. For example, in this tutorial you created a new notebook `panel-trees-dashboard.ipynb`, but you can also point to a Git repository.
5. Select the appropriate framework for your dashboard, in this example you'll have to select: `panel`.
6. Select the `conda` environment for your dashboard, make sure it is same as the one you previously selected as your Jupyter notebook environment
7. In the `relative path` box, copy your notebook's path (example: `demo-dashboards/tutorial/panel-trees-dashboard.ipynb`).
8. Once you have provided all the details above click on the save button.

You will then be redirected to a new window where you will be able to select the compute resources for your dashboard.

:::warning
The available compute instances might vary depending on the configuration and cloud provider of your Nebari instance.

Also, the **best instance type** for your dashboard will depend on your specific use case.
:::

An example of available compute instances available within a Nebari instance is shown in the following image:
![Nebari Instance selection UI screenshot for the Trees Dashboard. The radio button for the `Small instance - Stable environment with 1 CPU / 4 GB ram is selected](/img/tutorials/window_nebari_select_instance_type.png)

For this particular tutorial, a small instance should be enough. Once you have made a selection you can click on the **Save** button at the bottom of the window.
This will trigger the deployment of your dashboard, and you'll be presented with a screen displaying the of this process.

![Nebari window displaying the progress of the Trees' dashboard deployment. This window displays a message reading "The dashboard is starting up"](/img/tutorials/nebari_window_dashboard_starting_up.png)

If there are no errors encountered during this process, you will be automatically redirected to the dashboard!

</TabItem>
</Tabs>

## Manage apps in Nebari

<Tabs>
<TabItem value="jhub-apps" label="JHub App Launcher" default>

All applications are available on the Nebari home page (from JupyterLab, click on the Nebari logo in the top right corner or go to `File` -> `Home`).

To manage an application, click on the three dots in the top right of the corresponding application card where you can:

- **Start** the app is it's not running
- **Stop** a running app
- **Edit** the application details
- **Delete** the app

![](/img/tutorials/jhub-apps-manage-app.png)

</TabItem>
<TabItem value="cds-dashboards" label="CDS Dashboards (Nebari v2023.7.1 or earlier)">

In the Nebari home page (from JupyterLab, click on the `File` menu tab, then select `Hub Control Panel` to go to the home page), click on `Dashboards` in the top navigation bar.

This will redirect you to Nebari's Dashboard main panel.
Here, you can find the URL of your dashboards which can be shared with other users, as well as manage the status and deployment of your dashboards:

- To stop the dashboard server click on the `stop` button.
- To start the dashboard server click on the `start` button.
- To delete the server and the resources allocated click on the `delete` button.
- To make any changes to the existing CDS options, click on the `edit` button.

![Nebari dashboard panel - showing a number of dashboards with corresponding start/delete buttons, as well as several URLs under the "Dashboards from others heading"](/img/tutorials/nebari_dashboard_panel.png).

</TabItem>
</Tabs>

:::important
While the dashboard is running, it will continue to consume resources.
You should be mindful of the incurring ongoing costs while the dashboard is running, and stop it when not needed.
:::

---

Dashboards and apps can be very handy tools to share information and insights with colleagues and external customers or collaborators. You can use this basic dashboard to build more complex dashboards, add more dynamic features, and start sharing data insights with others.

<!-- Internal links -->

[login-keycloak]: /docs/tutorials/login-keycloak
[create-env]: /docs/tutorials/creating-new-environments
[jub-app-launcher]: /docs/how-tos/jhub-app-launcher



---
File: /nebari-docs/docs/docs/tutorials/creating-new-environments.md
---

---
id: creating-new-environments
title: Manage packages and environments
description: Using conda-store for environment management
---

Nebari uses [`conda-store`][conda-store-docs] for managing and sharing reproducible
environments (collection of specific packages and versions) on the platform.

:::warning
conda-store is the most reliable way to manage your packages and environments on
Nebari.

Refrain from installing libraries directly in the notebook or through the terminal
(outside a conda-store managed environment). These actions could lead to subtle
and unforeseen problems with your environment.
:::

## Preliminary reading

It's useful to understand basics of conda-store and how it builds on top of the
conda ecosystem, to use it effectively in Nebari.

- [Essential conda concepts: Packages, environments, channels, dependencies, etc.](https://conda.store/conda-store/explanations/conda-concepts)
- [conda-store concepts: Reproducibility, namespaces, environment versions, roles, etc.](https://conda.store/conda-store/explanations/conda-store-concepts)

Nebari has conda-store integrated, and you can use it through the graphical UI.

## Open `conda-store` web interface

![Conda Store WebUI interface](/img/tutorials/conda_store_webui.png)

There are several options for navigating to conda-store:

- From Nebari Home, click on **"Environment Management"** under "Services"
- From JupyterLab, click on `Nebari` in the menu bar and go to **"Environments"**
- From anywhere, go to URL: `https://<your-nebari-domain>/conda-store`

If not logged in to conda-store, click on the **"Log in"** button in th left
sidebar and authenticate similar to the [Nebari login][login-keycloak]. This is
required to be able to access many `conda-store` features.

## Create, edit, and manage environments

Go through the following conda-store (UI) tutorials
on using the graphical interface for various actions:

- [Create new environments][cs-create-env]
- [Edit & delete existing environments][cs-edit-delete-env]
- [Switch environment versions][version-control]

## Default namespaces in Nebari

A default Nebari deployment/instance has the following namespaces corresponding
to [Nebari groups][configure-keycloak-groups]:

<!-- Verify the roles and actions -->

- `analyst` namespace - Users in the `analyst` group can view and `admin` group
  can view+edit the environments in this namespace
- `developer` namespace - Users in the `developer` and `admin` groups can view+edit
  the environments in this namespace
- `nebari-git` namespace - Everyone can view and `admin`s can edit

As an individual user, you also have a personal namespace with the same name as
your Nebari username.

:::note
If you can "view" an environment, you can use it.
:::

## Select environments in editors

Instructions to select any environment you have access to in the following editing
spaces:

- **JupyterLab** - In a Jupyter Notebook, click on the "Select Kernel" dropdown in
  the top-left corner, and select the environment.

- **VS Code** - Click on the ⚙️ icon in the bottom-right to open `Settings` ->
  `Command Palette`, and type "Python: Select Interpreter" and press <kbd>Enter</kbd>
  to get the list of environments to select from.

- **Terminal** - In the terminal window, you can use `conda` CLI commands like
  `conda activate <namespace>-<environment_name>` to activate the relevant
  environment and `conda env list` to view the list of available environments.

## Special requirements

### Dask

Include the [`nebari-dask` metapackage](https://anaconda.org/conda-forge/nebari-dask)
in your environment to use Dask. This ensures you have the correct version of
`dask-gateway` and the latest versions of `dask` and `distributed` libraries.

By default, the `nebari-git-nebari-git-dask` environment (available to everyone)
can be used for basic Dask workflows.

### JHub App Launcher

Include the `jhub-apps` package in your environment to create apps using the
JHub App Launcher. You will also need the relevant app development framework
and other necessary packages in the environment.

<!-- External links -->

[conda-store-docs]: https://conda.store/
[cs-create-env]: https://conda.store/conda-store-ui/tutorials/create-envs
[cs-edit-delete-env]: https://conda.store/conda-store-ui/tutorials/edit-delete-envs
[version-control]: https://conda.store/conda-store-ui/tutorials/version-control

<!-- Internal links -->

<!--Update when PR#397 is merged -->

[login-keycloak]: /docs/tutorials/login-keycloak
[configure-keycloak-groups]: /docs/how-tos/configuring-keycloak#in-depth-look-at-roles-and-groups



---
File: /nebari-docs/docs/docs/tutorials/index.mdx
---

import DocCardList from '@theme/DocCardList';
import {useCurrentSidebarCategory} from '@docusaurus/theme-common';

# Nebari tutorials

<div align="center">
  <img
    src="/img/welcome/tutorials-icon.svg"
    width="30%"
    style={{border: "none", paddingBottom: 10 + "px", boxShadow: "none"}}
  />
</div>

The goal of these tutorials is to help you create a mental model for how Nebari works.

<DocCardList items={useCurrentSidebarCategory().items}/>



---
File: /nebari-docs/docs/docs/tutorials/login-with-keycloak.md
---

---
id: login-keycloak
title: Authenticate and launch JupyterLab
description: |
  A step-by-step guide to logging into your Nebari instance and launching a JupyterLab pod.
---

# Authenticate and launch JupyterLab

This guide provides a basic overview of how to log in to your Nebari instance (through Keycloak) and launch JupyterLab on a new server (ie., machine).

## 1. Connect to your Nebari instance

Navigate to the instance homepage.

Once on the site, you will be prompted to log in, similar to the login page shown in the image below. The text on this screen is configurable so your deployment may look slightly different.

![Nebari login screen](/img/how-tos/nebari_login_screen.png)

Nebari uses [Keycloak](https://www.keycloak.org/), an open-source identity and access management tool. Keycloak is a centralized location for administrators to add new users, create user groups and update roles. For more information, see the [How to Configure Keycloak][nebari-keycloak] docs.

:::warning
Nebari **does not** create a "regular" user for you during deployment, it only creates the `root` Keycloak admin user. The following steps assume that your Nebari administrator has already created a Keycloak user in the Keycloak admin console for you. If not, follow the steps in [Adding new users to Keycloak][nebari-keycloak-add-user] to create a Keycloak user.
:::

## 2. Authenticate with your provider

At this stage, it is a little challenging to provide thorough instructions because the particulars will depend on your [identity provider](https://www.keycloak.org/docs/latest/server_admin/#_identity_broker) / authentication provider ([LDAP](https://pt.wikipedia.org/wiki/LDAP), [OAuth 2.0](https://oauth.net/2/), passwordless authentication, password-based authentication and many others).

For more information on how to configure Keycloak, and add new users, review the [How to configure Keycloak][nebari-keycloak] sections of the docs.

In this example, Auth0 is the identity provider.
To continue the login process,
select the `Auth0` button and login.

<p align="center">
<img src="/img/tutorials/keycloak_nebari_login.png" alt="Nebari Keycloak auth screen - shows a form to provide username or password or to authenticate through Auth0" width="400"/>
</p>

## 3. Navigate the landing page

:::note
You may need to "Authorize access" to reach the home page:

<p align="center">
<img src="/img/tutorials/jhub-apps-authorize.png" alt="Authorize access page with an 'Authorize' button requesting access for JHub Apps, which is a Jupyter Service, to identify the user." width="80%"/>
</p>
:::

Once authenticated, the **Nebari home page** opens.
The home page, also referred to as
"Landing page", "JupyterHub launcher", or "App Launcher",
lists Nebari's core **services** and allows you to [create and share web apps][create-dashboard].

Click on the **"JupyterLab"** button under **"Quick Access"** to start a server and launch JupyterLab:

![Nebari home page with Services: JupyterLab, Argo, Users, Environments, Monitoring, VSCode; My Apps: JupyterLab (default JupyterLab instance); and Shared Apps.](/img/tutorials/nebari-home.png)

:::warning
This landing page was added recently and is in active development.
You can follow the progress on [JHub App Launcher's GitHub repository][jhub-apps-gh].

To go back to the legacy version, [disable the launcher in `nebari-config.yaml`][install-jhub-apps].
:::

### Launch a new server

Landing page to Launch server:
![Nebari dashboard main screen - displays a button "Launch Server"](/img/how-tos/nebari_main_launch_hub_page.png)

Click on the **Launch Server** button to select a profile and launch JupyterLab.

:::note
Nebari versions 2023.12.1 and earlier had a different landing page with core services listed in the top navigation bar:
![Nebari dashboard main screen - displays a button "Start my server"](/img/how-tos/nebari_main_hub_page.png)
In which case, click on the **Start My Server** button to select a profile and launch JupyterLab.
:::

## 4. Select a profile

The customized profiles (also called "instances", "servers" or "machines") will give you access to fixed cloud resources as mentioned in the list.

These options are configured by your administrator, learn more in the [Profile Configuration documentation][profile-configuration].

![Nebari select profile](/img/how-tos/nebari_select_profile.png)

Select an appropriate profile and click **"Start"**.

## 5. Start your server

After clicking "Start", your JupyterLab instance starts to launch.

This step may take up to several minutes due to Nebari's use of autoscaling under the hood. Ultimately this autoscaling feature helps reduce costs when the cluster is idle.

A progress bar is shown while the server is being set up:

![Nebari start server](/img/how-tos/nebari_server_start.png)

<details>
<summary>Optional: View Event log</summary>

During this time you might see some log messages that detail the autoscaling process. Click the **Event Log** button to view them:

![Nebari event log](/img/how-tos/keycloak_start_event_logs.png)

</details>

:::warning

- The starting up sequence can take up to several minutes, depending on the size of the cluster. If the server is not accessible **after 10 minutes**, an error will be shown, and you will be redirected to the Home page. Please check the [troubleshooting docs][troubleshooting] for more information.
- The Event logs may include warnings about `"[Warning] 0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector ..." `. This is **normal behaviour** as the cluster sometimes need to expand in order to start the server.
- Warning such as `"[Warning]" Unable to retrieve some image pull secrets ..."` can also be safely ignored.
  :::

Once your server is ready, you will be redirected to JupyterLab!

<!-- Internal links -->

[nebari-keycloak]: /docs/how-tos/configuring-keycloak
[nebari-keycloak-add-user]: /docs/how-tos/configuring-keycloak#adding-a-nebari-user
[install-jhub-apps]: /docs/how-tos/jhub-app-launcher#installation-on-nebari
[profile-configuration]: /docs/explanations/profile-configuration
[troubleshooting]: /docs/troubleshooting
[create-dashboard]: /docs/tutorials/create-dashboard

<!-- External links -->

[jhub-apps-gh]: https://github.com/nebari-dev/jhub-apps



---
File: /nebari-docs/docs/docs/tutorials/run-notebook-as-a-job.md
---

---
id: jupyter-scheduler
title: Submit notebook jobs with jupyter-scheduler
description: Use jupyter-scheduler to submit scheduled or unattended jobs
---

# Submit Notebook jobs with jupyter-scheduler

:::warning
This is a new feature still in beta so please [leave some feedback][feedback-link] and [report any issues][issues-link].

There is one known issue with the `Update Job Definition` and `Resume` job definition which is related to [Nebari-Workflow-Controller issue #18][workflow-controller-issue]. The current workaround for those who need to update (or pause) your job definitions, is simply to delete the current job definition and create a new one as and when needed.
:::

Some Nebari users require scheduling notebook runs. This is now achievable with [Jupyter-Scheduler][jupyter-scheduler], a JupyterLab extension that has been enhanced and integrated into Nebari. Additionally, users can monitor their job status by accessing the `<nebari-domain>/argo` endpoint.

Notebook jobs are useful in situations where you need no human interaction in the notebook and the results can be efficiently saved to your home directory, the cloud, or other storage locations. It is also useful in situations where the notebook might run for a long time and the user needs to shut down their JupyterLab server.

:::info
Shutting down your JupyterLab server (and associated JupyterLab extensions) doesn't stop your running Jupyter-Scheduler jobs because the Nebari team extended Jupyter-Scheduler to use Argo-Workflows as it's backend. Details can be found in the [argo-jupyter-scheduler repo][argo-jupyter-scheduler].

:::

Jupyter-Scheduler is included by default in the base Nebari JupyterLab image and can be used with any conda-store environment available to the notebook author (`papermill` must be included in the environment).

:::note
By default, only users in the `admin` or `developer` groups will have permission to create notebook jobs. For more information regarding users and groups, please visit [How to configure Keycloak][configure-keycloak]
:::

## Submitting a Notebook as a Jupyter-Scheduler Job

To submit your job, simply click the `Jupyter-Scheduler` icon on the top of your notebook toolbar.

![Jupyter-Scheduler UI - location of the icon on the notebook toolbar](/img/tutorials/jupyter-scheduler-icon.png)

This will open a new **Create Job** tab. Fill out the form and click "Create" to create a schedule.

<div align="center">
  <img src="/img/tutorials/jupyter-scheduler-run-now.png" alt="Jupyter-Scheduler UI - layout of Jupyter-Scheduler UI." width="60%"/>
</div>

:::note
Include [`papermill`][papermill] in your conda-store environment.
:::

Once created, the status of the job and the notebook output can be viewed from the Jupyter-Scheduler Notebook Jobs UI:

![Jupyter-Scheduler UI - view the notebook job status](/img/tutorials/jupyter-scheduler-job-status.png)

The output of your job will be accessible via the download icon in the "Output files" column. After you download the output, new links will replace the icon and point to the location of the downloaded output.

Click on the notebook job name to view more information about the job.

![Jupyter-Scheduler UI - view the notebook job details](/img/tutorials/jupyter-scheduler-job-details.png)

If you need to reopen the Notebook Jobs tab, look for the icon in the upper right of the JupyterLab launcher tab.

![JupyterLab-Launcher UI - location of Notebook Jobs icon](/img/tutorials/jupyterlab-launcher-notebook-jobs-icon.png)

:::info
As mentioned above, the notebook job will run as an Argo-Workflows workflow. This means the jobs (workflows) are also viewable from the Argo-Workflows UI at `<nebari-domain>/argo`. The name of the workflow is prefixed with `job-<job-id>`, which will match the notebook job ID.

![Argo UI - view the notebook job status from the argo UI](/img/tutorials/jupyter-scheduler-argo-ui.png)

All associated workflows have a TTL (time-to-live) set to 600 seconds. This means that regardless of whether or not your workflow was successful, it will be deleted after 10 mins from the Argo UI. The downside is that those logs are now gone. However, the upside is that deleted workflows no longer consume compute resources.
:::

## Submit Jupyter Notebooks to run on a schedule

The other way notebooks can be run is on a specified schedule, Jupyter-Scheduler refers to these as "Job Definitions". After selecting the Jupyter-Scheduler icon at the top of the notebook, select "Run on a schedule" and then define the schedule. There is also an option to specify the time zone.

:::tip
You can use [crontab.guru][crontab-guru], which is a nifty tool that tries to translate the schedule syntax into plain English.
:::

<div align="center">
  <img src="/img/tutorials/jupyter-scheduler-job-definition.png" alt="Jupyter-Scheduler UI - create a job definition to run on a specified schedule." width="60%"/>
</div>

When a job definition is created, a new job is created at each time interval specified by the schedule. These jobs can be inspected by clicking on "Notebook Job Definitions" near the top of the Jupyter-Scheduler UI. From here you have several options, including:

- **Delete** the job definition
- **Pause** the job job definition
- view details such as the **Status** of the job definition
- view the list of notebook jobs created from this job definition
- view the **Tags** (by default all job definitions are tagged with `Cron-Workflow`)

<div align="center">
  <img src="/img/tutorials/jupyter-scheduler-job-def-details.png" alt="Jupyter-Scheduler UI - view a job definition details." width="90%"/>
</div>

:::info
Unlike a regular notebook job, job definitions create Argo-Workflows cron-workflows (workflows that run on a schedule). This cron-workflow in turn creates workflows at each time interval specified by the schedule. Another difference is that the workflow names are prefixed with `job-def-<job-definition-id>`.

![Argo UI - view the job definition from the argo UI](/img/tutorials/jupyter-scheduler-argo-cron-workflow.png)
:::

:::warning
Notebook jobs that run on a schedule will run indefinitely so it's the responsibility of the job creator to either delete or pause the job if then they are no longer needed.
:::

## Debugging failed jobs

Even if your notebook runs normally in JupyterLab, it might fail during a scheduled run.

<div align="center">
  <img src="/img/tutorials/jupyter-scheduler-job-failed.png" alt="Jupyter-Scheduler UI - view the details of a failed job." width="90%"/>
</div>

You can view the job log and the notebook using either the terminal or the file browser (go to the JupyterLab View menu and enable "Show Hidden Files"). These are the steps to get the logs:

1. Find the job ID of your failed job
2. Launch a terminal session or go to your file browser
3. Navigate to `~/.local/share/jupyter/scheduler_staging_area`
4. Find the folder that corresponds to the job ID
5. Open (or `cat`) any log file or notebook files and look for error information

:::info
Notebook job details can also be viewed from the `<nebari-domain>/argo` UI if the job is still listed.
:::

Lastly, if the job fails without writing to the `scheduler_staging_area`, or the job status is stuck in `In progress` mode for an extended period of time, have an administrator try and view the specific logs on your JupyterLab server pod or on the workflow pod itself.

[feedback-link]: https://github.com/nebari-dev/nebari/discussions
[issues-link]: https://github.com/nebari-dev/nebari/issues
[workflow-controller-issue]: https://github.com/nebari-dev/nebari-workflow-controller/issues/18
[jupyter-scheduler]: https://jupyter-scheduler.readthedocs.io/en/latest/index.html#
[argo-jupyter-scheduler]: https://github.com/nebari-dev/argo-workflows-executor
[papermill]: https://papermill.readthedocs.io/en/latest/
[configure-keycloak]: /how-tos/configure-keycloak-howto.md
[crontab-guru]: https://crontab.guru



---
File: /nebari-docs/docs/docs/tutorials/using_dask.md
---

---
id: using_dask
title: Work with big data using Dask
description: Introduction to Dask with Nebari
---

# Work with big data using Dask

## Introduction

Working with large datasets can pose a few challenges - perhaps the most common is memory limitations on the
user's machine.

[Dask](https://www.dask.org/) is a flexible, open source library for parallel and distributed computing in Python.
Dask allows data scientists the ability able to scale analyses from a sample dataset to the full, large-scale dataset
with almost no code changes.
It's a powerful tool that can revolutionize how you do analytics!

## Dask integration on Nebari

Nebari uses [Dask Gateway][dask-gateway] to expose auto-scaling compute clusters automatically
configured for the user, and it provides a secure way to managing Dask clusters.

<details>
<summary> Click here for quick notes on how Dask works! </summary>

Dask consists of 3 main components `client`, `scheduler`, and `workers`.

- The end users interact with the `client`.
- The `scheduler` tracks metrics and coordinates workers.
- The `workers` have the threads/processes that execute computations.

The `client` interacts with both `scheduler` (sends instructions) and `workers` (collects results).

Check out the [Dask documentation][dask-docs] and the [Dask Gateway documentation][dask-gateway] for a full explanation.

</details>

## Step 1 - Set up Dask Gateway

Let's start with a fresh Jupyter notebook.
Select an environment from the `Select kernel` dropdown menu
(located on the top right of your notebook).
On a default Nebari deployment, you can select the `nebari-git-nebari-git-dask` environment.

:::warning
Be sure to select an environment which includes `Dask`, and note that the versions of `dask`, `distributed`, and `dask-gateway` must be the same.
We recommend the [`nebari-dask` metapackage](https://anaconda.org/conda-forge/nebari-dask). We publish this metapackage alongside Nebari to easily provide you with the correct Dask packages and versions, just be sure that the `nebari-dask` version matches your Nebari deployment version.
:::

Nebari has set of pre-defined options for configuring the Dask profiles that we have access to.
These can be accessed via Dask Gateway options.

```python
from dask_gateway import Gateway
# instantiate dask gateway
gateway = Gateway()

# view the cluster options UI
options = gateway.cluster_options()
options
```

![Nebari - Cluster Options UI](/img/tutorials/dask_cluster_options.png)

Using the `Cluster Options` interface, you can specify the `conda` environment for Dask workers, the instance type for the workers, and any additional
environment variables you'll need.

:::warning
It’s important that the environment used for your notebook matches the Dask worker environment!

The Dask worker environment is specified in your deployment directory under `/image/dask-worker/environment.yaml`
:::

:::tip
The fields displayed in the Cluster Options UI are defined in the `nebari-config.yml`.
By default, you see "Environment", "Cluster Profile", and "Environment Variables", but you can configure this as per your (and your team's) needs.
:::

## Step 2 - Create a Dask cluster

1. Let's start by creating a new Dask cluster from within your notebook:

   ```python
   # create a new cluster with our options
   cluster = gateway.new_cluster(options)
   # view the cluster UI
   cluster
   ```

   Once you run the cell, you'll see the following:
   ![Creating a Gateway Cluster UI](/img/tutorials/dask_cluster_creation.png)

2. You have the option to choose between `Manual Scaling` and `Adaptive Scaling`.

   If you know the resources that would be required for the computation, you can select `Manual Scaling` and
   define a number of workers to spin up. These will remain in the cluster until it is shut down.

   Alternatively, if you aren't sure how many workers you'll need, or if parts of your workflow require more workers
   than others, you can select `Adaptive Scaling`. Dask Gateway will automatically scale the number of workers
   (spinning up new workers or shutting down unused ones) depending on the computational border. `Adaptive Scaling` is
   a safe way to prevent running out of memory, while not burning excess resources.

   For this tutorial, we suggest trying the Adaptive Scaling feature.
   In the Gateway Cluster UI, set the minimum value to 1 worker and maximum to 5 workers, and click on the `Adapt` button.

   You may also notice a link to the Dask Dashboard in this interface. We'll discuss this in a later section.

## Step 3 - View the Dask Client

Once the Dask cluster has been created, you'll be able to get the cluster's information directly from your Jupyter notebook:

```python
# get the client for the cluster
client = cluster.get_client()
# view the client UI
client
```

On executing the cell, you'll see the following:
![Nebari - Dask client UI showing details of the newly created Dask cluster](/img/tutorials/dask_client.png)

The `Dask Client` interface gives us a brief summary of everything we've set up so far.

## Step 4 - Understand Dask's diagnostic tools

Dask comes with an inbuilt dashboard containing multiple plots and tables containing live information as
the data gets processed.

:::note
To use the dashboard for the first time, click on the dashboard link displayed in the Client UI.
This opens a familiar Keycloak authentication page, where you can [sign-in the same way you authenticated into Nebari](/docs/tutorials/login-keycloak).
Dask's browser-based dashboard opens automatically.
:::

You can access the dashboard in two ways:

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<Tabs>
  <TabItem value="browser-based" label="Browser-based diagnostic dashboard" default>

Dask's diagnostic dashboard will open in a new browser tab if you click on the dashboard links displayed in the Client UI or Cluster UI above.

![Dask diagnostic UI - showing four parallel computation streams](/img/tutorials/dask_diagnostic_dashboard.png)

  </TabItem>
  <TabItem value="labextension" label="Dask JupyterLab extension (recommended)">

The [Dask-labextension](https://github.com/dask/dask-labextension) is included with the default Nebari deployment.
It provides a JupyterLab extension to manage Dask clusters,
as well as embed Dask's dashboard plots directly into JupyterLab panes.

On JupyterLab, you can select the Dask icon in the left sidebar and click on the magnifying glass icon to view the list of available plots.
Alternatively, you can copy-and-paste the dashboard link (displayed in the Client UI) in the input field.

![Dask-labextension UI In JupyterLab - displays the demo notebook and a list of cluster settings](/img/tutorials/dask_labextension.png)

  </TabItem>
</Tabs>

Let's open and understand the dashboard plots `Task Stream`, `Progress`, and `Cluster map`.

Most colors and the interpretation would differ based on the computation you choose.
However, the colors remain consistent and represent the same task across all plots within the dashboard.
Note that some colors always have the same meaning. For example, red always indicates inter-worker communication and data transfer.

Each of the computations you submit to Dask (you will learn more in the next section) is split into multiple tasks for parallel execution.

In the `Progress` plot, the distinct colors are associated with different tasks to complete the overall computation.

In the `Task Stream` (a streaming plot) each row represents a thread (a Dask worker can have multiple threads). The small rectangles within are the individual tasks running over time.

The `Cluster Map` shows the Dask scheduler at the center in a purple circle, with the active Dask workers around it in yellow circles.
This diagram is a convenient way to visualize Adaptive Scaling, and ensure new workers are being spun-up and shut-down based on workflow demand.

Check out the [Dask Documentation on the dashboard plots](https://docs.dask.org/en/stable/dashboard.html) for more information.
Keep the dashboard plots open for the following computations!

## Step 5 - Run your computation with Dask

In the following sections, you will perform some basic analysis on the well-known [New York City yellow taxi dataset](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page),
a subset of which we have copied over to a Google Cloud Storage bucket `gs://nebari-public/yellow_taxi_NYC`.

:::note
This dataset is saved in Parquet format, a column-oriented file format commonly used for large datasets saved in the cloud.
:::

1. To get started, we will load the data using Dask's Dask DataFrame API. The following will lazily load the dataset::

   ```python
   import dask.dataframe as dd

   ddf = dd.read_parquet(
       path='gs://nebari-public/yellow_taxi_NYC/*/*.parquet',
       storage_options={'anon':True},
   )
   ```

2. From here you can start analyzing the data. First, let's check the size of the overall dataset:

   ```python
   dataset_size = ddf.memory_usage(deep=True).compute().sum()
   print(dataset_size / 10**9, "GB")
   ```

   **Output:**

   ```shell
   32.42624498 GB
   ```

   This corresponds to 32.43 GB of data. Running this one-liner would be impossible on most single machines but running this on a Dask cluster with 4 workers, this can be calculated in under a minute.

3. Now, let's perform some actual analysis! We can for example, compare the number of taxi rides from before, during and after the COVID-19 pandemic.
   To do this, you'll need to aggregate the number of rides per day, calculate a 7-day rolling average and then compare these numbers for the same day (April 15th) across three different years, 2019, 2020, and 2022:

   ```python
   # get the pickup date, ignoring pickup time
   ddf["pickup_date"] = ddf.tpep_pickup_datetime.dt.date.astype(str)
   # aggregate rides by pickup date
   gb_date = ddf.groupby(by="pickup_date").agg("count")
   # calculate a 7-day rolling average of the number of taxi rides
   gb_date["num_rides_7_rolling_ave"] = gb_date.tpep_pickup_datetime.rolling(7).mean()
   ```

4. Now you can compare number of taxi rides on April 15th across three different years. In the `Cluster Map`, notice how new workers are spun-up to execute that task.

   **For April 15th, 2019 - pre-pandemic**

   ```python
   gb_date.loc["2019-04-15"].num_rides_7_rolling_ave.compute()
   ```

   **Output:**

   ```shell
   pickup_date
   2019-04-15    259901.142857
   Name: num_rides_7_rolling_ave, dtype: float64
   ```

   **or April 15th, 2020 - during the height of the pandemic**

   ```python
   gb_date.loc["2020-04-15"].num_rides_7_rolling_ave.compute()
   ```

   **Output:**

   ```shell
   pickup_date
   2020-04-15    7161.857143
   Name: num_rides_7_rolling_ave, dtype: float64
   ```

   **For April 15th, 2022 - post-pandemic**

   ```python
   gb_date.loc["2022-04-15"].num_rides_7_rolling_ave.compute()
   ```

   **Output:**

   ```shell
   pickup_date
   2022-04-15    119405.142857
   Name: num_rides_7_rolling_ave, dtype: float64
   ```

   There were about 260,000 taxi rides a day in middle of April 2019 and that number plummeted to over 7,100 rides a year later, a full two orders of magnitude fewer riders. Wild!

Performing this kind of analysis on such a large dataset would not be possible without a tool like Dask. On Nebari, Dask comes out-of-the-box ready to help you handle these larger-than-memory (out-of-core) datasets.

## Step 5 - Shutdown the cluster

As will you have noticed, you can spin up a lot of compute really quickly using Dask.

**With great power comes great responsibility**

Remember to shut down your cluster once you are done, otherwise this will be running in the background, and you might incur unplanned costs.
You can do this from your Jupyter notebook:

```python
cluster.close(shutdown=True)
```

## Using Dask safely

If you're anything like us, we've forgotten to shut down our cluster a time or two. Wrapping the `dask-gateway` tasks in a
context manager is a great practice that ensures the cluster is fully shutdown once the task is complete!

### Sample Dask `context manager` configuration

You can make use of a context manager to help you manager your clusters. It can be written once and
included in your codebase. Here is a basic example to get you started (which can then be extended to suit your needs):

```python title="Sample Dask example using context manager"
import os
import time
import dask.array as da
from contextlib import contextmanager

import dask
from distributed import Client
from dask_gateway import Gateway

@contextmanager
def dask_cluster(n_workers=2, worker_type="Small Worker", conda_env="nebari-git/nebari-git-dask"):
    try:
        gateway = Gateway()
        options = gateway.cluster_options()
        options.conda_environment = conda_env
        options.profile = worker_type
        print(f"Gateway: {gateway}")
        for key, value in options.items():
            print(f"{key} : {value}")

        cluster = gateway.new_cluster(options)
        client = Client(cluster)
        if os.getenv("JUPYTERHUB_SERVICE_PREFIX"):
            print(cluster.dashboard_link)

        cluster.scale(n_workers)
        client.wait_for_workers(1)

        yield client

    finally:
        cluster.close()
        client.close()
        del client
        del cluster
```

Now you can write your compute tasks inside the context manager and all the setup and teardown will be managed for you:

```python
with dask_cluster() as client:
    x = da.random.random((10000, 10000), chunks=(1000, 1000))
    y = x + x.T
    z = y[::2, 5000:].mean(axis=1)
    result = z.compute()
    print(client.run(os.getpid))
```

Kudos, you should now have a working Dask cluster inside Nebari.
Now go load up your own big data!

<!-- Reusable links -->

[dask-website]: https://www.dask.org/
[dask-docs]: https://docs.dask.org/en/stable/
[dask-gateway]: https://gateway.dask.org/



---
File: /nebari-docs/docs/docs/faq.md
---

# Frequently asked questions

## Why is the `NEBARI_KUBECONFIG` file in `/tmp`?

Nebari regenerates this file on every run. This means it will be removed by the operating system during its cleanup process,
but running the `nebari deploy` command again as a Nebari administrator will update/create a `NEBARI_KUBECONFIG` file for you.

## How are conda user environments created? Who creates them?

`Conda-store` manages all environments on the Nebari. It allows users to create private environments in their own namespace, or shared environments under a group namespace. For more details check out the doc on [creating environments in Nebari via conda-store](/docs/tutorials/creating-new-environments).

Additionally, there is a legacy approach which is still available in Nebari. Administrators can create global environments by specifying the environment in `nebari-config.yml`. Environments specified in this way will be made available for all users and services under the `nebari-git` namespace on `conda-store`.

## What should be included in the environment if I want to use Dask?

There are drop-in replacements for `distributed`, `dask`, and `dask-gateway` with the correct pinned versions available via the [Nebari Dask metapackage](https://github.com/conda-forge/nebari-dask-feedstock). Example: `nebari-dask==||nebari_VERSION||`.

## How can I install a package locally? Will this package be available to Dask workers?

If you want to install a package locally, we suggest following the guide for [developing local packages][developing-packages].

It's important to note that packages installed this way aren't available to the Dask workers. See our [Dask tutorial][dask-tutorial] for more information.

## Can I modify the `.bashrc` file on Nebari?

Nebari automatically creates and manages `.bashrc` and `.profile`. Therefore, end users do not have write permissions to modify this file. However, by default Nebari _will_ source `.bash_profile`. Users may use this file to populate environment variables or set up alias, etc. However, there are some important things to note:

- The `.bash_profile` is sourced _after_ the `.bashrc` - be aware of the implications, one of which is that you will lose changes to the prompt syntax. To avoid this, you can always source the `.bashrc` inside the .`bash_profile`.
- JupyterLab kernels do _not_ source `.bash_profile` but the Jupyter terminal does.
- The VS Code terminal does _not_ source `.bash_profile` by default.

## What if I can't see the active conda environment in the terminal?

Set the `changeps1` value in the `conda` config:

```shell
conda config --set changeps1 true
```

The `conda` config is located in the `/home/{user}/.condarc` file. You can change the conda config with a text editor (for example: `nano`, which is included in Nebari by default), and the changes will be applied on saving the file.

## How do I clean up old environment builds in conda-store?

You may find that the pods hosting your environment get full over time, prompting you to clear them out. As an admin, you can delete environments completely (including all builds of the environment) in the conda-store UI. Go to the environment, click `Edit` and then click `Delete`.

If you'd like to retain the latest version of an environment and only remove specific builds, you'll need to navigate to the conda-store admin page located at `<nebari-domain/conda-store/admin>`. Click on the environment you'd like to clean up. At the bottom of the page, there is a list of each environment build, each with it's own "delete" button.

## How do I use preemptible and spot instances on Nebari?

A preemptible or spot VM is an instance that you can create and run at a much lower price than normal instances. Azure
and Google Cloud platform use the term preemptible, while AWS uses the term spot.
However, the cloud provider might stop these instances if it requires access to those
resources for other tasks. Preemptible instances are excess Cloud Provider's capacity, so their availability varies with
usage.

#### Usage

##### Google Cloud Platform

The `preemptible` flag in the Nebari config file defines the preemptible instances.

```yaml
google_cloud_platform:
  project: project-name
  region: us-central1
  zone: us-central1-c
  availability_zones:
  - us-central1-c
  kubernetes_version: 1.18.16-gke.502
  node_groups:
# ...
    preemptible-instance-group:
      preemptible: true
      instance: "e2-standard-8"
      min_nodes: 0
      max_nodes: 10
```

##### Amazon Web Services

Spot instances aren't supported at this moment.

##### Azure

Preemptible instances aren't supported at this moment.

## Why doesn't my code recognize the GPU(s) on Nebari?

First be sure you chose a [GPU-enabled server when you selected a profile][selecting a profile]. Next, if you're using PyTorch, see [Using GPUs on Nebari][using gpus]. If it's still not working for you, be sure your environment includes a GPU-specific version of either PyTorch or TensorFlow, i.e. `pytorch-gpu` or `tensorflow-gpu`. Also note that `tensorflow>=2` includes both CPU and GPU capabilities, but if the GPU is still not recognized by the library, try removing `tensorflow` from your environment and adding `tensorflow-gpu` instead.

## How do I migrate from Qhub to Nebari?

Nebari was previously called QHub. If your Qhub version lives in the `0.4.x` series, you can migrate to Nebari by following the [migration guide](./how-tos/nebari-upgrade). If you're using a version of Qhub that lives in the `0.3.x` series, you will need to upgrade to `0.4.x` first as the user group management is different between the two versions. For more information, see the deprecation notice in the [Nebari release note](./references/RELEASE).

## Why is there duplication in names of environments?

The default Dask environment is named `nebari-git-nebari-git-dask`, with `nebari-git` duplicated.

`nebari-git` is the name of the namespace.
Namespaces are a concept in conda-store, however conda itself does not recognize it.

It is possible to use conda-store to create an environment with the name "dask" in two different namespaces.
But because conda doesn't understand namespaces, conda won't be able to differentiate between them.
To avoid this, we prepend the namespace's name into the environment building on conda-store.

Next, [nb_conda_kernels](https://github.com/Anaconda-Platform/nb_conda_kernels) with [nb-conda-store-kernels](https://pypi.org/project/nb-conda-store-kernels/) are the packages that we use to transform conda environments into runnable kernels in JupyterLab (that's why we require that all environments have `ipykernel`).

The issue is that `nb_conda_kernels` insists the following path: `/a/path/to/global/datascience-env`, which corresponds to `global-datascience-env` being the name that users see while `datascience-env` is what conda sees.

Hence, to make things unique we've named things as `/a/path/to/global/global-datascience-env`. This makes conda see the env as `global-datascience-env`, but `nb_conda_kernel` now displays it as `global-global-datascience-env`.

We have discussed contributing a PR to `nb_conda_kernels`, but the project has not accepted community PRs in over 3 years, so we don't currently have the motivation to do this.

If you have potential solutions or can help us move forward with updates to the `nb_conda_kernels`, please reach out to us on our [discussion forum](https://github.com/orgs/nebari-dev/discussions)!

## Why does my VS Code server continue to run even after I've been idle for a long time?

Nebari automatically shuts down servers when users are idle, as described in Nebari's documentation for the [idle culler settings][idle-culler-settings]. This functionality currently applies only to JupyterLab servers. A VS Code instance, however, runs on Code Server, which isn't managed by the idle culler. VS Code, and other non-JupyterLab services, will not be automatically shut down.
:::note
Until this issue is addressed, we recommend manually shutting down your VS Code server when it is not in use.
:::

<!-- Internal links -->

[dask-tutorial]: tutorials/using_dask.md
[idle-culler-settings]: https://www.nebari.dev/docs/how-tos/idle-culling
[selecting a profile]: tutorials/login-keycloak#4-select-a-profile
[using gpus]: how-tos/use-gpus
[developing-packages]: how-tos/develop-local-packages



---
File: /nebari-docs/docs/docs/glossary.md
---

# Nebari Glossary



---
File: /nebari-docs/docs/docs/troubleshooting.mdx
---

---
title: Troubleshooting
description: Guidance for troubleshooting common issues in Nebari.
---

# Troubleshooting

Invariably you will encounter behavior that does not match your expectations.
This guide is meant to explain that behavior, give you some context around
why it's happening, help you diagnose it, and if possible resolve it.

## Configuration

### Delete a conda-store build, environment or namespace

:::note
Please be patient as conda-store deletes the build, environment or namespace as it can take up to 30 minutes for these changes to propagate.
:::

#### Delete a conda-store build

If you wish to delete a conda-store build, your user will need the `conda_store_admin` (in the `admin` group) or `conda_store_superadmin`
(in the `superadmin` group) role to be perform this action.

1. Navigate to https://your-nebari-domain/conda-store/admin/ and login
2. Navigate to the appropriate conda-store namespace and environment
3. Locate the build you wish to delete and then click the trash can icon to delete the build

<div class="text--center">
  <img src="/img/troubleshooting/delete_build.png" width={420} />
</div>

#### Delete a conda-store environment

If you wish to delete a conda-store environment, your user will need the `conda_store_admin` (in the `admin` group) or `conda_store_superadmin`
(in the `superadmin` group) role to be perform this action.

The steps to delete an environment are practically the same:

1. Navigate to https://your-nebari-domain/conda-store/admin/ and login
2. Navigate to the appropriate conda-store namespace and environment
3. When ready, click on the "delete" button

<div class="text--center">
  <img src="/img/troubleshooting/delete_environment.png" width={420} />
</div>

#### Delete a conda-store namespace

If you wish to delete a conda-store namespace, your user will need the `conda_store_superadmin` (in the `superadmin` group) role to be perform this action.

1. Navigate to https://your-nebari-domain/conda-store/admin/user and login
2. Click on the "Manage namespaces" button

<div class="text--center">
  <img src="/img/troubleshooting/manage_namespaces.png" width={420} />
</div>

3. Find the namespace you wish to delete and click on the "delete" button

<div class="text--center">
  <img src="/img/troubleshooting/delete_namespace.png" width={420} />
</div>

:::warning
Please avoid deleting the `nebari-git` namespace as it is used by the base JupyterLab image and deleting it will likely cause JupyterLab to stop working!
:::

### Conda-store compatibility migration steps when upgrading to 0.4.5

If you are upgrading from a version of Nebari prior to `0.4.5`, you will need to manually update your conda-store namespaces
to be compatible with the new Nebari version. This is a one-time migration step that will need to be performed after upgrading to continue using the service.

After upgrading to `0.4.5`, the base names for the following namespaces that are built in with Nebari were changed as follows:

- `default` -> `global`
- `filesystem` -> `nebari-git`

Due to this change, you will need to migrate the environments from the old `default` and `filesystem` namespaces into the new `global` and `nebari-git` respectively.
This is a manual process that consists in copying over the yaml specifications for each environment to the new namespaces and rebuilding them.

Once all environments are migrated, you will be able to delete the `default` and `filesystem` namespaces using the `delete` option from the conda-store UI:

![delete-namespace](/img/troubleshooting/delete_namespace_045_upgrade.png)

:::warning
Both the `filesystem` and `nebari-git` are built-in namespaces that are used by Nebari.
Deleting or modifying these namespaces will require special permission levels that are not granted by default.
Follow the instructions in the [Handle Access to restricted namespaces](troubleshooting#handle-access-to-restricted-namespaces) to grant the necessary permissions.
:::

### Required pins for Dask environments

There are some pins that are required for the Nebari Dask environment to function correctly.

The best way to manage Dask pins is to use the `nebari-dask`
[metapackage on conda-forge](https://anaconda.org/conda-forge/nebari-dask).
Usage will look something like this:

```yaml
environments:
  environment-dask.yaml:
    name: dask
    channels:
    - conda-forge
    dependencies:
    - python
    - ipykernel
    - ipywidgets
    - nebari-dask ==0.2.3
```

The pins for the metapackage can be found in the [conda-forge recipe](https://github.com/conda-forge/nebari-dask-meta-feedstock/blob/master/recipe/meta.yaml).

### SSL certificate issues with ClearML

If you are using ClearML and you are getting SSL certificate errors, this might be due to a mismatch between the certificate used by ClearML and the one used by Nebari.
This normally happens when a custom certificate was provided to Nebari that does not englobe the ClearML domains, which are:

```
- files.clearml.<nebari-domain>
- app.clearml.<nebari-domain>
- api.clearml.<nebari-domain>
```

If the Domain Name System (DNS) and Common Name (CN) name doesn't match, Traefik then generates and uses a self-signed certificate.
Though as it's not possible to request a double wildcard certificate for a domain (for example \*.\*.local.com). This may lead to some unexpected [TLS](https://www.internetsociety.org/deploy360/tls/basics) issues,
so as alternative to including each specific domain under the certificate CN list, you may also define a wildcard certificate.
See [default certificate configuration](https://doc.traefik.io/traefik/https/tls/#default-certificate) for more details.

## Errors

### Default conda-store environment fails to build on initial deployment

One of the two conda-store environments created during the initial Nebari deployment (`dashboard` or `dask`) may fail to appear as options when logged into JupyterHub.

If your user has access to conda-store, you can verify this by visiting `<your_nebari_domain>.com/conda-store` and having a look at the build status of the missing environment.

The reason for this issue is how these environments are simultaneously built.
Under the hood, conda-store relies on Mamba/Conda to resolve and download the specific packages listed in the environment YAML.
If both environment builds try to download the same package with different versions, the build that started first will have their package overwritten by the second build.
This causes the first build to fail.

To resolve this issue, navigate to `<your_nebari_domain>.com/conda-store`, find the environment build that failed and trigger it to re-build.

### DNS `domain={{ your_nebari_domain }}` record does not exist

During your initial Nebari deployment, at the end of the `04-kubernetes-ingress` stage, you may receive an output message stating that the DNS record for `your_nebari_domain` "appears not to exist, has recently been updated, or has yet to fully propagate."

As the output message indicates, this is likely the result of the non-deterministic behavior of the DNS.

Without going into a deep dive of what DNS is or how it works, the issue encountered here is that when Nebari tries to look up the IP address associated with the DNS record, `your_nebari_domain`, nothing is returned. Unfortunately, this "lookup" is not as straightforward as it sounds. To lookup the correct IP associated with this domain, many intermediate servers (root, top level domain, and authoritative nameservers) are checked, each with their own cache, and each cache has its own update schedule (usually on the order of minutes, but not always).

As the output message mentions, it will ask if you want it to retry this DNS lookup again after another wait period; this wait period keeps increasing after each retry. However, it's possible that after waiting 15 or more minutes that the DNS still won't resolve.

At this point, feel free to cancel the deployment and rerun the same deployment command again in an hour or two. Although not guaranteed, it's likely that the DNS will resolve correctly after this prolonged wait period.

If you are interested in learning more about DNS, [this "How DNS works comic" by dnsimple](https://howdns.works/) is a great resource to get started.

### New Dask Gateway clusters lead to `500 Internal Server Error`

This may occur when `dask-gateway-controller` has some internal authentication issues. To resolve this, you can restart `nebari-daskgateway-controller` and `nebari-daskgateway-controller` pods:

- Open `k9s` (Learn more about `k9s` in [Debug Nebari deployment](./how-tos/debug-nebari#debugging-the-kubernetes-cluster-using-k9s))
- Press the <kbd>/</kbd> key and search for "daskgateway"
- Use the arrow keys to highlight each pod
- Delete highlighted pods with <kbd>ctrl</kbd> + <kbd>d</kbd>, and the pods should restart automatically

### Conda-store storage threshold

Nebari has a predefined (and configurable) storage limit for private and shared conda environments created via `conda-store`. A common pitfall is having the created environments fill up the available space, resulting in the following error in the conda-store UI when trying to [create new environments or builds](https://www.nebari.dev/docs/tutorials/creating-new-environments):

> `CondaStore.storage_threshold` reached

To remove unused or unwanted environments and free up some space, you can follow the [Delete a conda-store build, environment or namespace](https://www.nebari.dev/docs/troubleshooting#delete-a-conda-store-build-environment-or-namespace) steps.

While this process does indeed free up space in the storage drive, currently trying to create a new environment or build using the conda-store UI keeps failing because it does not re-scan available space automatically. In order to have conda-store re-scan the available space and allow the creation of new environments and builds, the conda-store worker pod must be restarted.

You can do this via either `k9s` or `kubectl`.

If you prefer `kubectl`, simply run:

```bash
kubectl delete pods -l role=nebari-conda-store-worker -n dev
```

Make sure to change `dev` if you specified a different namespace in the configuration.

If you prefer `k9s`:

- Open `k9s` (Learn more about `k9s` in [Debug Nebari deployment](./how-tos/debug-nebari#debugging-the-kubernetes-cluster-using-k9s))
- Press the <kbd>/</kbd> key and search for "nebari-conda-store-worker"
- Use the arrow keys to highlight each pod
- Delete highlighted pods with <kbd>ctrl</kbd> + <kbd>d</kbd>

In both cases, a new conda-store worker pod will be immediately restarted, and you will be able to create new environments and builds again.

## How do I...?

### Get the Kubernetes context

Depending on a variety of factors, `kubectl` may not be able to access your Kubernetes cluster.
To configure this utility for access, depending on your cloud provider:

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<Tabs>
  <TabItem value="gcp" label="Google Cloud" default>

Check the [Google Cloud Platform documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl) for more information.

1. Download the [GCP SDK](https://cloud.google.com/sdk/downloads).
2. Initialize the gcloud CLI with `gcloud init`. This will prompt you to log in to your GCP account.

3. Fetch your cluster credentials with:

```bash
gcloud container clusters \
  get-credentials "<your-cluster-name>" \
  --region "<region>"
```

</TabItem>

<TabItem value="aws" label="Amazon AWS">

Check the [Amazon Web Services documentation](https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html) for more information.

1. Download the [AWS CLI](https://aws.amazon.com/cli/).
2. If you haven't already, [create an AWS Access Key and Secret Key](https://aws.amazon.com/premiumsupport/knowledge-center/create-access-key/).
3. Create a `kubeconfig` file with:

```bash
aws eks update-kubeconfig --region "<region-code>" \
  --name "<your-cluster-name>"
```

</TabItem>
</Tabs>

Once you have created a `kubeconfig` file `kubectl` should be able to access your Kubernetes cluster.

### Deploy an arbitrary pod

A straightforward way to deploy arbitrary pods would be to use `kubectl apply` on the pod manifest.

### Upgrade the instance size for the `general` node group

The `general` node group, or node pool, is (usually) the node that hosts most of the pods that Nebari relies on for its core services: `hub`, `conda-store`, `proxy` and so on.
The instance by default, follows a "min-max" approach to determine the instance size: large enough so that the initial deployment will work out of the box,
while keeping total cloud compute costs to a minimum.

Although each cloud provider has different names and hourly prices for their compute nodes, the default `general` node group in `nebari-config.yaml` has 2 vCPU and 8 GB of memory.

Based on testing, clusters running on Google Kubernetes Engine (GKE) appear to be amenable to in-place upgrades of the `general` node instance size.
Unfortunately, this does not seem to be the case with the other cloud providers, and attempting to do so for AWS and Azure will likely result in the catastrophic destruction of your cluster.

| Cloud Provider | `general` node upgrade possible? |
| :------------- | :------------------------------- |
| AWS            | No (Danger!)                     |
| Azure          | No (Danger!)                     |
| GCP            | Yes                              |

If modifying the resource allocation for the `general` node in-place is absolutely necessary, try increasing the maximum number of nodes for the `general` node group.
This will mean two nodes (reserved for the `general` node group) will always be running, ultimately increasing the operating cost of the cluster.

:::warning
Given the possible destructive nature of resizing this node group, it is **highly recommended** to back up your cluster before attempting this.
:::

Alternatively, you can backup your cluster, destroy it, specify the new instance size in your `nebari-config.yaml`, and redeploy.

### Use a DNS provider other than CloudFlare

CloudFlare is one of the most commonly used DNS providers for Nebari, so to some it may seem as if it is the _only_ DNS provider Nebari supports.
This is **not** the case.

The [Setup Nebari domain registry](./how-tos/domain-registry) section in these docs contain detailed instructions on how to change or set a new DNS provider.

### Add system packages to a user's JupyterLab image

In some cases, you may wish to customize the default user's JupyterLab image, such as installing some system packages via apt (or other OS package manager) or adding some JupyterLab extensions.

Nebari uses its own registered docker images for `jupyterhub`, `dask`, and `jupyterlab` services by default, but this can be changed by:

1.  Building your own docker images, and
2.  Including the DockerHub register hash into `nebari-config.yaml`.

### Provide individual users with unique environments and cloud instance types

Nebari allows for admins to set up user groups which each have access to specific environments and server instance types. This allows for a fine-tuned management of your cloud resources. For example, you can create a special user group for a team of ML engineers which provides access to GPUs and a PyTorch environment. This will prevent inexperienced users from accidentally consuming expensive resources. Provided you have performed some setup ahead of time, users can choose both instance types as well as environments at server launch time.

First, you will need to create new node groups, one for each type of GPU instance you would like to provide users.

Second, you will need to create a new JupyterLab profile to select the GPU node.
In this way, you'll have a separate profile for GPUs which users can select.

### Migrate existing environments from a namespaces to another

When you create a new environment as a user, it requires selecting a namespace. This namespace is used to identify the group permissions to
that environment and the location where the environment needs to be stored.
To migrate environments from one namespace to another, you must manually build all environments in the new namespace.
Once migrated, you can delete the old namespace, permanently deleting all existing environments built against it.

### Handle Access to restricted namespaces

The built in namespace `nebari-git` is used to build the environments from the `nebari-config.yaml` file.
As its use is reserved for conda-store server, and is directly controlled by Nebari, modifying or deleting the contents of such namespaces without system permission level
will result in a permission denied message from conda-store.

As the conda-store system permission level is not granted to any user on Nebari, to modify this namespace, you will need to manually update the `conda-store-config.py` configmap located in the
kubernetes cluster to lift the restriction. Follow the steps below to do so: (`kubectl` or `k9s` are required)

With `kubectl`:

```bash
kubectl edit configmap conda-store-config -n nebari
```

Once the editor opens, look for the `role_bindings` map and update
the `default_namespace` value from `{"viewer"}` to `{"admin"}`. And restart the conda-store
pods to fully propagate the changes.

With `k9s`:

1.  Type <kbd>:configmap</kbd> to list the available configmaps.
2.  Search for `conda-store-config` using the shortcut <kbd>/conda-store-config</kbd>.
3.  When hovering over the `configmap`, press <kbd>e</kbd> to open the editor. (defaults to `vim`)
4.  Look for the `role_bindings` map and update the `default_namespace` value from `{"viewer"}` to `{"admin"}`.
5.  Save and exit the editor. (`:wq`)
6.  Restart the conda-store pods to fully propagate the changes.

:::warning
Based on the Nebari and conda-store security model, the `default_namespace` value should be set to `{"viewer"}` after the migration is complete. Or followed by a re-deployment of Nebari
to restore the default value.
:::



---
File: /nebari-docs/docs/docs/welcome.mdx
---

---
title: Welcome to Nebari's documentation
slug: /welcome
---

Nebari is an open source data science platform, developed for collaboration and scalability.
It aims to provide a robust infrastructure that can quickly be set up (deployed).

![Nebari overview sequence showing deploying from yaml to cloud resources to the instance spinup selection](/img/welcome/nebari_overview_sequence.png)

## Learning paths

Nebari's individual documentation page have specific purpose and an intended audience.
You can be an administrator configuring resources for your team or a data scientist looking to create dashboards (or you might alternate among roles over time).

The following tabs provide a quick guide for the different paths you can take to navigate Nebari's documentation.

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<Tabs>
  <TabItem value="data-practitioner" label="Data practitioner" default>

If you are a data practitioner using Nebari for your day-to-day work, like data analysis, visualization, and more, the following path and pages will be useful for you.

🔸 [Log-in to your team's Nebari instance, launch your JupyterLab server](/docs/tutorials/login-keycloak), and use the IPython/Jupyter notebooks.

🔸 Learn to [create new environments (kernels) using conda-store](/docs/tutorials/creating-new-environments) if you need specific libraries.

🔸 If you need to scale your work, [leverage Dask and Dask Gateway for parallel and distributed computing within Nebari][using-dask].

🔸 Learn to [create and share interactive data visualization dashboards with your team][create-dashboards].


🔸 You can also [consider Visual Studio Code][using-vscode] if that is more convenient for your work.

  </TabItem>

  <TabItem value="administrator" label="Administrator">

If you are setting up Nebari and managing the platform for your organization or team, the following steps and pages will be helpful.

🔸 Begin with the ["Get Started" documentation section][get-started] that includes complete instructions to install and deploy your own Nebari instance.

🔸 The get started section will also take you to some "how-to guides" like the [deployment instructions for specific cloud-providers](/docs/get-started/cloud-providers) and [DNS provisioning](/docs/how-tos/domain-registry).

🔸 After setup, [provide your team members access to your Nebari instance by configuring Keycloak][configure-keycloak].

🔸 If you face issues or challenges, check out the [Troubleshooting section](/docs/troubleshooting) and [GitHub Discussions][github-discussions] to learn from the community. You can also look into [debugging your deployment](/docs/how-tos/debug-nebari).

🔸 As a good security practice, [create and keep a manual backup](/docs/how-tos/manual-backup) of your configurations and your team's work.

🔸 To get the latest features, bug fixes, and security patches, [upgrade your Nebari instance](/docs/how-tos/nebari-upgrade) regularly.

🔸 If or when you need to, you can [destroy the instance and all associated resources](/docs/how-tos/nebari-destroy).

  </TabItem>

  <TabItem value="contributor" label="Contributor">

Thanks for your interest in contributing to Nebari! Nebari is a fully open source project, and relies on contributors like you to continue growing.

🔸 Learn all the different ways in which you can [participate in the community, like reporting bugs, contributing with and without code][community].

🔸 Read the [release notes][release-notes] to stay informed about Nebari's latest improvements.

  </TabItem>

  <TabItem value="other" label="Just curious">

Welcome curious person! Glad your're interested in Nebari!
Take a look at the home page at [nebari.dev][nebari-home] for an overview of Nebari.

To learn more, you can briefly go through the pages in the ["Getting started" section][get-started], then check out the pages listed in the "Data Scientist" tab here!

If you have any questions, you can reach the Nebari community on [GitHub Discussions][github-discussions].

  </TabItem>
</Tabs>

## Get Started

New to Nebari? Start here to set up your own Nebari instance.

> [Learn how to install and deploy Nebari from scratch →](/docs/get-started/installing-nebari)

> Refer to [CLI commands in the Quick start →](/docs/get-started/quickstart)

<!-- TODO: All a link to Nebari 101 when it's ready. -->

[↳ Read all the getting started resources][get-started]

<img
  src="/img/welcome/tutorials-icon.svg"
  width="15%"
  align="right"
  style={{border: "none", boxShadow: "none"}}
/>

## Tutorials

Learn to use different Nebari features and integrations with our end-to-end, standalone walkthroughs.

> [Work with big data using Dask →][using-dask]

> [Create, deploy, and share a Panel dashboard →][create-dashboards]

[↳ Read all tutorials][tutorials]

<img
  src="/img/welcome/how-tos-icon.svg"
  width="20%"
  align="right"
  style={{border: "none", boxShadow: "none"}}
/>

## How-to guides

Learn to perform specific tasks in Nebari, with step-by-step instructions.

> If you're an administrator, read [how-to configure Keycloack for all users →][configure-keycloak]

> If you're a data scientist, [learn how-to use Visual Studio code for your work →][using-vscode]

[↳ Read all guides][how-tos]

<!-- TODO: Add conceptual guides when ready

<img
  src="/img/welcome/conceptual-guides-icon.svg"
  width="20%"
  align="right"
  style={{border: "none", boxShadow: "none"}}
/>

## Conceptual guides

> TBD

[↳ Read all guides](./) -->

<img
  src="/img/welcome/community-icon.svg"
  width="16%"
  align="right"
  style={{border: "none", boxShadow: "none"}}
/>

## Community

Nebari is a community-led open source project, learn how you can get involved!

> [Facing issues with Nebari? Get support on GitHub discussions or the issue tracker →](/community/introduction#getting-support)

> [Learn how you can contribute to Nebari, beyond code and pull-requests →](/community/introduction#how-to-contribute)

[↳ Read all community resources][community]

<img
  src="/img/welcome/references-icon.svg"
  width="18%"
  align="right"
  style={{border: "none", boxShadow: "none"}}
/>

## References

<!-- TODO: Add API doc when ready -->

Understand technical details of Nebari's architecture and how it works.

> [Release notes →][release-notes]

[↳ Read all references][references]

<!-- External links -->

[github-discussions]: https://github.com/orgs/nebari-dev/discussions

<!-- Internal links -->

[nebari-home]: https://nebari.dev/
[get-started]: /docs/get-started/
[tutorials]: /docs/tutorials/
[how-tos]: /docs/how-tos/
[community]: /community/introduction
[references]: /docs/references
[release-notes]: /docs/references/RELEASE
[using-dask]: /docs/tutorials/using_dask
[using-vscode]: /docs/how-tos/using-vscode
[configure-keycloak]: /docs/how-tos/configuring-keycloak
[create-dashboards]: /docs/tutorials/create-dashboard
