Index

Outline
Introduction
Jupyter Notebook on an AWS Instance
Tensorboard on AWS Instance
File Transfer
Tmux: Terminal Multiplexer
Managing Virtual Environments
Integrating VSCode with AWS EC2
Automating Instance Stoppage
Conclusion

Outline

Introduction

Briefly explain the importance of mastering AWS for efficient cloud computing.
Mention the goal of the blog: to provide essential AWS tricks and tips for optimizing your workflow.

1. Jupyter Notebook on an AWS Instance

A step-by-step guide to setting up a Jupyter Notebook on an AWS EC2 instance.
- Launching an instance and selecting the appropriate configuration.
- Connecting to the instance using SSH and setting up the required environment.
- Running Jupyter Notebook and accessing it on your local browser.
Highlight the benefits of using Jupyter Notebook on AWS for data analysis and machine learning tasks.

2. Tensorboard on AWS Instance

Explaining how to utilize Tensorboard to visualize TensorFlow model performance on an AWS instance.
Connecting to the instance and running Tensorboard.
Accessing Tensorboard through your local browser for real-time model monitoring.

3. File Transfer

Demonstrating how to transfer files between your local machine and an AWS EC2 instance using scp.
Step-by-step instructions for both uploading files to the instance and downloading files to your local machine.

4. Tmux: Terminal Multiplexer

Introducing Tmux and its advantages for remote working.
Installing Tmux on your AWS instance and creating a new session.
Running processes in the background and reattaching to existing sessions.

5. Managing Virtual Environments

Setting up virtual environments on an AWS EC2 instance for better project isolation.
Creating and activating virtual environments with different Python versions.
Managing multiple environments efficiently to avoid version conflicts.

6. Integrating VSCode with AWS EC2

Explaining how to integrate Visual Studio Code with your AWS EC2 instance for a seamless coding experience.
Troubleshooting connectivity issues and optimizing the development environment.

7. Automating Instance Stoppage

Configuring an alarm to automatically stop an AWS EC2 instance when it becomes idle based on specific conditions.
Preventing unwanted automatic stoppage when the action is triggered.

Conclusion

Summarize the key AWS tricks and tips covered in the blog.
Encourage readers to implement these techniques to enhance their productivity and proficiency in using AWS.

Introduction

Welcome to the world of Amazon Web Services (AWS) tricks and tips! AWS has revolutionized the way we deploy, scale, and manage our applications in the cloud. Whether you are a seasoned AWS professional or just starting with cloud computing, this blog aims to provide you with valuable insights and techniques to supercharge your AWS workflow.

As developers, data scientists, and engineers, we encounter various challenges when working with AWS services. From setting up Jupyter Notebooks on EC2 instances to transferring files and managing virtual environments, every task can be more efficient with the right approach. That's where this blog comes in – we'll explore a collection of practical tricks and tips that will empower you to work smarter and more effectively with AWS.

So, whether you're looking to improve your development environment, streamline file management, or automate tasks for cost-saving benefits, this blog has you covered. Let's dive in and discover how you can make the most out of AWS and take your cloud computing skills to the next level.

Jupyter Notebook on an AWS Instance

Jupyter Notebook is a powerful tool for interactive data analysis, machine learning experimentation, and collaborative coding. By running Jupyter Notebook on an AWS EC2 instance, you can leverage the computing resources of the cloud to handle resource-intensive tasks efficiently. In this section, we will walk you through the process of setting up and accessing Jupyter Notebook on an AWS instance.

Step 1: Launching the AWS EC2 Instance

Open your browser and navigate to the Amazon AWS Console (aws.amazon.com/console).
Sign in to your AWS account or create one if you haven't already.
In the AWS Management Console, go to the EC2 dashboard.

Step 2: Launching the EC2 Instance

Click on the "Launch Instance" button to start the instance creation process.
Choose the "Deep Learning AMI GPU PyTorch" (let us say) as the Amazon Machine Image (AMI) to get pre-installed GPU drivers and libraries for deep learning tasks.
Select the "g4dn.xlarge" (let us say) instance type. This instance type provides a balance of compute power and memory suitable for most machine learning workloads.
Select your preferred Key Pair or create one to connect to the instance securely.

Step 3: Configuring Network Settings

Under Network Settings, change the "Allow SSH traffic from" drop-down box to "My IP." This ensures that only your IP address is allowed to connect to the instance via SSH, enhancing security.

Step 4: Launching the Instance

Click "Launch Instance" to start the provisioning process for your chosen instance type and settings.
Select "Connect to instance" to access the newly launched EC2 instance.

Step 5: Connecting to the Instance via SSH

Open "Git Bash" on your Windows machine or your terminal on a macOS/Linux machine.
Move to the directory where the Key Pair (.pem file) is located using the cd command.
Set the appropriate permissions for the Key Pair file to ensure secure access:
```
 chmod 400 your-key-pair.pem
```
Run the following command in the terminal to connect to your EC2 instance:
```
 ssh -i your-key-pair.pem ec2-user@your-ec2-public-ip
```

Replace your-key-pair.pem with the name of your Key Pair file, and your-ec2-public-ip with the public IP address of your EC2 instance.

Step 6: Activating the Required Environment

Once you are connected to the EC2 instance via SSH, activate the required environment by referring to the welcome message. The environment might be named "pytorch" in this case:
```
 source activate pytorch
```

Step 7: Starting Jupyter Notebook

Update the package lists to ensure you have the latest versions of the installed software:
```
 sudo apt update
```
Launch Jupyter Notebook without starting a browser session using the --no-browser flag:
```
 jupyter notebook --no-browser
```

Note: You may be required to install the necessary packages.

Step 8: Accessing Jupyter Notebook on Your Local Browser

Open another terminal window on your local machine (Windows: Git Bash, macOS/Linux: Terminal).
Move to the directory where your Key Pair (.pem file) is located using the cd command.
Run the following command with port forwarding to access Jupyter Notebook securely:
```
 ssh -NfL 9999:localhost:8888 -i your-key-pair.pem ec2-user@your-ec2-public-ip
```
Replace your-key-pair.pem with the name of your Key Pair file, and your-ec2-public-ip with the public IP address of your EC2 instance.
Open your web browser and go to http://localhost:9999/. You should see the Jupyter Notebook interface, and you can start working on your projects.

Running Jupyter Notebook on an AWS EC2 instance allows you to harness the computational power of the cloud while enjoying the familiar Jupyter environment. This setup is especially beneficial for data scientists, researchers, and machine learning practitioners working on resource-intensive tasks.

Tensorboard on AWS Instance

Tensorboard is a powerful visualization tool provided by TensorFlow that allows you to monitor and analyze the training process and performance of your machine-learning models. Running Tensorboard on an AWS EC2 instance enables you to visualize your models remotely and gain insights into their behavior. In this section, we will guide you through the process of setting up and accessing Tensorboard on an AWS instance.

Step 1: Connect to Your AWS Instance via SSH

Step 2: Starting Tensorboard

1. Once connected to your AWS instance via SSH, you can start Tensorboard using the following command:
```
  tensorboard --logdir=/path/to/logs/directory
```
  Replace /path/to/logs/directory with the directory path where your TensorFlow logs are stored. Tensorboard will read these logs and display the corresponding visualizations.

Step 3: Accessing Tensorboard on Your Local Browser

Open another terminal window on your local machine (Windows: Git Bash, macOS/Linux: Terminal).
Move to the directory where your Key Pair (.pem file) is located using the cd command.
Run the following command with port forwarding to access Tensorboard securely:
```
 ssh -NfL 6006:localhost:6006 -i your-key-pair.pem ec2-user@your-ec2-public-ip
```
Replace your-key-pair.pem with the name of your Key Pair file, and your-ec2-public-ip with the public IP address of your EC2 instance.
Open your web browser and go to http://localhost:6006/. You should see the Tensorboard interface, which provides various visualizations related to your model's training progress and evaluation metrics.

Running Tensorboard on an AWS EC2 instance facilitates remote model monitoring and analysis, making it convenient for machine learning practitioners and researchers to gain valuable insights into their models' performance.

File Transfer

Transferring files between your local machine and an AWS EC2 instance is a common task when working with cloud-based development environments. In this section, we will explore how to efficiently transfer files to and from an AWS EC2 instance using the scp command, making it seamless to manage your project files in the cloud.

Transferring Files From the Local Machine to EC2

Before transferring files, make sure you have the necessary AWS EC2 instance running and accessible via SSH using your Key Pair.
If you have multiple files to transfer, consider creating a zip archive to simplify the process:
```
 zip -r dataFile.zip /path/to/your/files
```
Replace /path/to/your/files with the directory containing the files you want to transfer.
Open your terminal or "Git Bash" (for Windows users) and navigate to the directory where your Key Pair (.pem file) is located using the cd command.
Use the scp command to securely transfer the files to your EC2 instance:
```
 scp -i your-key-pair.pem dataFile.zip ec2-user@your-ec2-public-ip:~/
```
Replace your-key-pair.pem with the name of your Key Pair file, dataFile.zip with the name of your zip archive (if applicable), and your-ec2-public-ip with the public IP address of your EC2 instance.

Transferring Files From EC2 to Local Machine

Open your terminal or "Git Bash" (for Windows users) and navigate to the directory where your Key Pair (.pem file) is located using the cd command.
To transfer a file from the EC2 instance to your local machine, use the scp command in the following format:
```
 scp -i your-key-pair.pem ec2-user@your-ec2-public-ip:/path/to/file ~/Desktop/target
```
Replace your-key-pair.pem with the name of your Key Pair file, your-ec2-public-ip with the public IP address of your EC2 instance, /path/to/file with the path of the file you want to transfer, and ~/Desktop/target with the destination path on your local machine.

Transferring Multiple Files or Directories

For transferring multiple files or entire directories, use the recursive (-r) option:
```
 scp -i your-key-pair.pem -r ec2-user@your-ec2-public-ip:/path/to/source ~/Desktop/target
```
Replace your-key-pair.pem with the name of your Key Pair file, your-ec2-public-ip with the public IP address of your EC2 instance, /path/to/source with the source directory you want to transfer, and ~/Desktop/target with the destination path on your local machine.

Efficient file transfer between your local machine and an AWS EC2 instance using the scp command streamlines your development workflow, allowing you to seamlessly work with your project files in the cloud.

Tmux: Terminal Multiplexer

Tmux is a powerful tool that enhances your productivity when working in a terminal environment. It allows you to create and manage multiple sessions, windows, and panes within a single terminal window. By using Tmux on your AWS EC2 instance, you can keep your processes running even if you get disconnected or close the terminal window. In this section, we will walk you through the setup and usage of Tmux on AWS for a more efficient and resilient terminal experience.

Step 1: Installing Tmux

To install Tmux on your AWS EC2 instance, open your terminal and update the package lists:
```
 sudo apt-get update
```
Then, install Tmux using the package manager:
```
 sudo apt-get install tmux
```

Step 2: Creating a New Tmux Session

To create a new Tmux session, simply run the following command:
```
 tmux new -s mysession
```
Replace mysession with your desired session name. This will start a new Tmux session with a single window.

Step 3: Running a Process in the Background

Within the Tmux session, you can run your desired process or script as usual. For example, to run a Python script:
```
 nohup python3 your_script.py &
```
This will run the script in the background, allowing you to continue working within the Tmux session.

Step 4: Detaching from a Tmux Session

To detach from the Tmux session without terminating the running processes, press Ctrl + B, then release and press D. This will bring you back to your regular terminal, leaving the Tmux session running in the background.

Step 5: Listing and Reconnecting to Tmux Sessions

To list all existing Tmux sessions, use the following command:
```
 tmux list-sessions
```
This will display a list of all active Tmux sessions along with their names.
To reconnect to a specific Tmux session, use the following command:
```
 tmux attach -t mysession
```
Replace mysession with the name of the session you want to reconnect to. This will bring you back to the Tmux session with all your running processes intact.

Step 6: Closing a Tmux Session

To close a Tmux session, first, attach to the session:
```
 tmux attach -t mysession
```
Inside the Tmux session, press Ctrl + B, then release and press :. This will bring up the Tmux command prompt at the bottom of the screen.
Type kill-session and press Enter. This will terminate the Tmux session along with all its running processes.

Tmux is an indispensable tool for managing terminal sessions effectively, especially when working on remote AWS instances. It ensures that your processes continue running even if the connection is interrupted, providing a seamless and resilient development experience.

Managing Virtual Environments

Virtual environments are a crucial aspect of Python development, allowing you to create isolated and self-contained environments for different projects. On an AWS EC2 instance, managing virtual environments becomes essential to avoid version conflicts and ensure that your projects run smoothly. In this section, we will explore how to set up and manage virtual environments on AWS to streamline your Python development.

Step 1: Installing Virtual Environment

To create virtual environments on your AWS EC2 instance, you need to install the virtualenv package. Use the following command to install it:
```
 pip install virtualenv
```

Step 2: Creating a New Virtual Environment

Navigate to the directory where you want to create your virtual environment. For example, if you are inside your project directory, run:
```
 cd /path/to/your/project
```
Create a new virtual environment with a specific Python version (e.g., Python 3.8) by running:
```
 virtualenv -p python3.8 myenv
```
Replace myenv with the name you want to give to your virtual environment.

Step 3: Activating the Virtual Environment

To activate the virtual environment, use the following command:
```
 source myenv/bin/activate
```
Replace myenv with the name of your virtual environment.
Once activated, your shell prompt will change, indicating that you are now inside the virtual environment. This means any Python packages you install will be confined to this environment, preventing conflicts with other projects.

Step 4: Deactivating the Virtual Environment

To exit the virtual environment and return to the system's global Python environment, use the deactivate command:
```
 deactivate
```

Step 5: Listing All Virtual Environments

To view all virtual environments created on your AWS instance, navigate to the directory where virtual environments are stored. By default, they are usually located in the .virtualenvs directory in your home folder:
```
 ls -l ~/.virtualenvs
```
The output will display the names of all your virtual environments.

Managing virtual environments on AWS EC2 instances enables you to work on different projects with different Python dependencies in isolation. This practice helps ensure that your projects remain robust and maintainable, even when using shared cloud resources.

Integrating VSCode with AWS EC2

Integrating Visual Studio Code (VSCode) with your AWS EC2 instance provides a convenient and feature-rich development environment, allowing you to code, debug, and manage your projects directly on the cloud. In this section, we'll guide you through the steps to set up and configure VSCode to seamlessly work with your AWS EC2 instance.

Step 1: Prerequisites

Make sure you have Visual Studio Code installed on your local machine. If you don't have it, you can download and install it from the official website (code.visualstudio.com).
Ensure that you have already launched an AWS EC2 instance and have the necessary SSH Key Pair to access it securely.

Step 2: Install the Remote - SSH Extension

Open Visual Studio Code on your local machine.
Click on the "Extensions" icon in the Activity Bar on the side of the window (or press Ctrl+Shift+X).
In the Extensions view search box, type "Remote - SSH."
Install the "Remote - SSH" extension provided by Microsoft. This extension allows you to work with remote environments, including SSH connections to AWS EC2 instances.

Step 3: Configure the config File

The config file is where you specify the SSH configurations for connecting to your remote AWS EC2 instance. By default, this file is located in the .ssh directory in your home folder.
To access the config file, open a terminal on your local machine (Command Prompt on Windows or Terminal on macOS/Linux).
Navigate to the .ssh directory:
```
 cd ~/.ssh
```
If the config file does not exist, create one:
```
 touch config
```
Open the config file in a text editor:
```
 code config
```
In the config file, add the following configuration for your AWS EC2 instance:
```
 Host my-ec2-instance
     HostName your-ec2-public-ip
     User ec2-user
     IdentityFile /path/to/your/keypair.pem
```
Replace my-ec2-instance with a custom name for your EC2 instance, your-ec2-public-ip with the public IP address of your EC2 instance, ec2-user with the appropriate SSH username for your instance, and /path/to/your/keypair.pem with the local path to your SSH Key Pair (.pem file).

Step 4: Connect to Your AWS EC2 Instance

Click on the "Remote Explorer" icon in the Activity Bar on the side of the window. This will open the Remote Explorer panel.
Click on the "SSH Targets" icon in the Remote Explorer panel and select "Add New SSH Host..."
In the input box that appears, enter the following:
```
 ssh -i /path/to/your/keypair.pem ec2-user@your-ec2-public-ip
```
Replace /path/to/your/keypair.pem with the local path to your SSH Key Pair (.pem file), and your-ec2-public-ip with the public IP address of your AWS EC2 instance.
Press Enter, and VSCode will attempt to connect to your AWS EC2 instance.

Step 5: Start Coding on AWS EC2

Once connected, you can start coding on your AWS EC2 instance directly from Visual Studio Code. All the files and directories on the remote instance will be accessible through File Explorer.
Use the integrated terminal to run commands on the remote server, just like you would in a local development environment.
You can also use VSCode's built-in Git integration to manage version control for your projects hosted on the AWS EC2 instance.

Integrating Visual Studio Code with your AWS EC2 instance brings the power of a full-featured development environment to the cloud, making it easy to work on your projects from anywhere while leveraging the computing resources of AWS.

Automating Instance Stoppage

AWS EC2 instances are billed based on their running time, so it's essential to optimize their usage to control costs effectively. One way to achieve this is by automating the stoppage of your AWS EC2 instance when it becomes idle based on specific constraints. In this section, we'll show you how to set up an automatic stoppage mechanism to ensure your instance stops when it's not actively used.

Step 1: Create an Alarm

Open the Amazon EC2 console and navigate to the "Instances" page.
Select the instance for which you want to set up the automatic stoppage.
Click on the "Monitoring" tab and choose "Create Alarm."
In the "Create Alarm" wizard, configure the alarm based on the condition that defines when the instance is considered idle. For example, you can set the condition as "CPUUtilization <= 10% for 3 consecutive periods of 5 minutes."
For the alarm action, choose "Stop" so that the instance will stop when the condition is met.
Review and create the alarm.

Step 2: Preventing Unwanted Stoppage

Once the alarm is created, the instance will transition from "OK" to "Alarm" state when the specified condition is met. The automatic stop action will then be triggered.
However, to prevent unwanted automatic stoppage, turn off the action of the alarm when you need to work on the instance. This can be achieved by disabling the alarm action through the Amazon CloudWatch console.

Step 3: Using Tmux for Background Processing (Optional)

To continue running background processes on the instance, even after detaching from the terminal, you can use Tmux as described in the "Tmux: Terminal Multiplexer" section of this blog. Tmux allows processes to run independently in the background, even when the terminal session is detached.
With Tmux, you can detach from the EC2 instance while your background processes continue running. You can later reconnect to the instance and reattach to the Tmux session to check the status of your processes.

By automating the stoppage of your AWS EC2 instance when it becomes idle, you can efficiently manage costs and ensure that your resources are utilized optimally.

Conclusion

Congratulations, you've successfully delved into a collection of valuable AWS tricks and tips! We hope this blog has provided you with the knowledge and tools to optimize your AWS workflow and make your cloud computing experience more productive and cost-effective.

From running Jupyter Notebooks on EC2 instances for data analysis to seamlessly transferring files and managing virtual environments, you now have a toolkit of practical techniques to tackle various challenges in your AWS projects.

By integrating Visual Studio Code (VSCode) with your AWS EC2 instance, you can take advantage of a full-featured development environment in the cloud. Additionally, automating instance stoppage when idle ensures you maintain control over your AWS costs and resources, allowing for efficient usage of cloud computing.

Remember, AWS is an ever-evolving platform, and there are always new tricks and tips to discover. Keep exploring, experimenting, and embracing best practices to stay ahead in the cloud computing game.

Thank you for joining us on this AWS journey. We hope these insights will serve as valuable assets in your cloud computing endeavors. Happy coding in the cloud!

About the Author:

Hiii, I'm Avinash, a recent computer science graduate. I am currently working as a Machine Learning Intern at Augrade.

Connect me through :

Feel free to correct me !! :)
Thank you folks for reading. Happy Learning !!! 😊

AWS Tricks and Tips: Streamlining Your Workflow for Productivity

Avinash R - July 22, 2023