Skip links

How to set up and run Llama3 on Windows

Jump To Section

How to set up and run Llama3 on Windows


A step-by-step guide to running this revolutionary AI model on Windows!

As a fellow AI enthusiast, I’ve been experimenting with various models and frameworks for months, including Fabric from Daniel Miessler. This open-source framework is designed to augment human capabilities using AI, providing a modular approach to solving specific problems. Worth checking out here!

In addition to Fabric, I’ve also been utilizing Ollama to run LLMs locally and the Open Web UI for a ChatGPT-like web front-end. This has allowed me to tap into the power of AI and create innovative applications.

What is Llama3 and how does it compare to its predecessor?

Recently, I stumbled upon Llama3. Developed by Meta, this cutting-edge language model boasts state-of-the-art performance and a context window of 8,000 tokens – double that of its predecessor, Llama2! The Llama3 family of models includes both pre-trained and instruction-tuned generative text models in 8 and 70B sizes.

The Llama3 instruction-tuned models are specifically designed for dialogue use cases and outperform many open-source chat models on industry benchmarks. What’s more, the model has been carefully optimized for helpfulness and safety – a crucial consideration when working with AI.

Are you ready to unleash the full potential of Large Language Models (LLMs) and experience the future of artificial intelligence? If so, then join me on this exciting journey as we explore how to set up and run Llama3 on Windows! 


How do I set up Llama3 on Windows? Follow these steps:

Download Llama3: Get the latest version of Llama3 from here.
– Install Hugging Face Transformers: You’ll need this library to run Llama3. Install it using pip: pip install transformers.
– Set up your environment: Make sure you have Python 3.8 or higher installed, along with the required dependencies (e.g., nltk, spaCy).
– Run Llama3: Use the following command to load and run Llama3: python -m transformers.LLama3.

That’s it! With these simple steps, you’ll be ready to unleash the power of Llama3 on your Windows machine.

What is Open WebUI?

Open WebUI is an extensible, feature-rich, and user-friendly self-hosted web interface that operates entirely offline. It supports various Large Language Model (LLM) runners, including Ollama and OpenAI-compatible APIs.

In this blog, I’ll be going over how to run Ollama with the Open-WebUI to have a ChatGPT-like experience without having to rely solely on the command line or terminal.

Below are the steps to install and use the Open-WebUI with llama3 local LLM

1. Install Ollama

For Windows

  • Download the installer here
  • Right-click on the downloaded OllamaSetup.exe file and select “Run as administrator”
image 3

1. The screenshot above displays the download page for Ollama.

For Linux WSL:

NOTE: For this method, make sure to have Ubuntu WSL installed, we can easily do this by visiting the Microsoft store and downloading the latest version of Ubuntu WSL. 

You must enable virtualization and Windows Subsystem for Linux in Windows Features. 

image 4

2. The screenshot above displays the option to enable Windows features.

2. Install Open-WebUI or LM Studio

Check out the Open WebUI documentation here 

image 5

3. The screenshot above displays the GitHub page for Open-WebUI.

Check out the LM Studio documentation and download LM Studio from here

image 6

4. The screenshot above displays the download page for LM Studio.

# Run the following command in PowerShell or Ubuntu WSL

docker run -d -p 3000:80 -e OLLAMA_BASE_URL=https://example.com -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main
image 8

5. The screenshot above exhibits the docker run command in PowerShell.

Install Docker Desktop by following the link here 

image 9

6. The screenshot above displays the download page for Docker.

Note- For Ubuntu WSL:

You must also enable WSL Integration into Docker Desktop to be able to run Open-WebUI with llama3 if you are running from WSL.

image 10

7. The screenshot above exhibits the WSL Integration into Docker Desktop.

3. Sign in to Open WebUI

Access Open WebUI at `http://localhost:3000` or `http://WSL_IP_HERE:3000`.
For the first time, you need to register by clicking “Sign up”.

image 11

8. The screenshot above displays the home screen for Open WebUI

4. Pull a model from Ollama.com

Click the settings icon in the upper right corner of Open WebUI and enter the model tag (e.g., `llama3`).
Click the download button on the right to start downloading the model.

image 12

9. The screenshot above displays the settings for Open WebUI to download llama3.

5. Select a model and enjoy your AI chat

image 13

10. The screenshot above displays the downloaded llama3 module.

Here is an example of what you can do with llama3 and Open-WebUI

  1. You can request a Disaster Recovery Plan outline and get step-by-step instructions to formulate a cohesive and efficient plan.
image 14

11. The screenshot above displays the response generated by llama3

2. Additionally, you can upload a document and summarize key points:

image 15

12. The screenshot above displays the llama3’s response to the uploaded document

Conclusion

Having local Large Language Models (LLMs) offers significant advantages. Local LLMs allow you to fine-tune your models using custom datasets, tailoring them to your unique objectives and tasks. This personalization ensures that your LLM aligns with your specific requirements, providing more relevant and accurate assistance.

Unlock the Power of Local Large Language Models: A Game-Changer for Data Security, Customization, and Innovation

As we’ve explored throughout this journey, having local Large Language Models (LLMs) is more than just a nicety – it’s a game-changer. By keeping your queries locally, you’re not only ensuring the security of your data but also gaining unparalleled control over your models.
Customization: The Key to Unlocking Your Unique Potential

Local LLMs also empower you to fine-tune your models using custom datasets, tailing your unique objectives and tasks.

Data Security: The Ultimate Peace of Mind

Imagine being able to ask questions and receive answers without worrying about sensitive information falling into the wrong hands. With local LLMs, that’s exactly what you get. By processing queries locally, you eliminate the risk of data privacy concerns and potential information leakage – a crucial consideration in today’s digital landscape.
I hope this guide has provided you with a comprehensive understanding of the benefits and possibilities of running local LLMs, setting you on a path to explore and harness their capabilities effectively.

As LLMs continue to advance, with models like Llama 3 pushing the boundaries of performance, the possibilities for local LLM applications are vast. From specialized research and analysis to task automation and beyond, the potential applications are limitless. With the ability to run LLMs locally, you gain control over your data and model customization, empowering you to leverage the full potential of these powerful language models while maintaining privacy and aligning with your unique needs.

Additional Resources

Fabric 

Ollama Web-UI 

Llama3 

Picture of Aladdin Elston

Aladdin Elston

Latest Reads

Subscribe

Suggested Reading

Ready to Unlock Your Enterprise's Full Potential?

Vikas Krishan

Chief Digital Business Officer and Head of the EMEA region

Vikas (Vik) Krishan serves as the Chief Digital Business Officer and Head of the EMEA region for Altimetrik. He is responsible for leading and growing the company’s presence across new and existing client relationships within the region.

Vik is a seasoned executive and brings over 25 years of global experience in Financial Services, Digital, Management Consulting, Pre- and Post-deal services and large/ strategic transformational programmes, gained in a variety of senior global leadership roles at firms such as Globant, HCL, Wipro, Logica and EDS and started his career within Investment Banking. He has developed significant cross industry experience across a wide variety of verticals, with a particular focus on working with and advising the C-Suite of Financial Institutions, Private Equity firms and FinTech’s on strategy and growth, operational excellence, performance improvement and digital adoption.

He has served as the engagement lead on multiple global transactions to enable the orchestration of business, technology, and operational change to drive growth and client retention.

Vik, who is based in London, serves as a trustee for the Burma Star Memorial Fund, is a keen photographer and an avid sportsman.

Megan Farrell Herrmanns

Chief Digital Officer, US Central

Megan is a senior business executive with a passion for empowering customers to reach their highest potential. She has depth and breadth of experience working across large enterprise and commercial customers, and across technical and industry domains. With a track record of driving measurable results, she develops trusted relationships with client executives to drive organizational growth, unlock business value, and internalize the use of digital business as a differentiator.

At Altimetrik, Megan is responsible for expanding client relationships and developing new business opportunities in the US Central region. Her focus is on digital business and utilizing her experience to create high growth opportunities for clients. Moreover, she leads the company’s efforts in cultivating and enhancing our partnership with Salesforce, strategically positioning our business to capitalize on new business opportunities.

Prior to Altimetrik, Megan spent 10 years leading Customer Success at Salesforce, helping customers maximize the value of their investments across their technology stack. Prior to Salesforce, Megan spent over 15 years with Accenture, leading large transformational projects for enterprise customers.

Megan earned a Bachelor of Science in Mechanical Engineering from Marquette University. Beyond work, Megan enjoys playing sand volleyball, traveling, watching her kids soccer games, and is actively involved in a philanthropy (Advisory Council for Cradles to Crayons).

Adaptive Clinical Trial Designs: Modify trials based on interim results for faster identification of effective drugs.Identify effective drugs faster with data analytics and machine learning algorithms to analyze interim trial results and modify.
Real-World Evidence (RWE) Integration: Supplement trial data with real-world insights for drug effectiveness and safety.Supplement trial data with real-world insights for drug effectiveness and safety.
Biomarker Identification and Validation: Validate biomarkers predicting treatment response for targeted therapies.Utilize bioinformatics and computational biology to validate biomarkers predicting treatment response for targeted therapies.
Collaborative Clinical Research Networks: Establish networks for better patient recruitment and data sharing.Leverage cloud-based platforms and collaborative software to establish networks for better patient recruitment and data sharing.
Master Protocols and Basket Trials: Evaluate multiple drugs in one trial for efficient drug development.Implement electronic data capture systems and digital platforms to efficiently manage and evaluate multiple drugs or drug combinations within a single trial, enabling more streamlined drug development
Remote and Decentralized Trials: Embrace virtual trials for broader patient participation.Embrace telemedicine, virtual monitoring, and digital health tools to conduct remote and decentralized trials, allowing patients to participate from home and reducing the need for frequent in-person visits
Patient-Centric Trials: Design trials with patient needs in mind for better recruitment and retention.Develop patient-centric mobile apps and web portals that provide trial information, virtual support groups, and patient-reported outcome tracking to enhance patient engagement, recruitment, and retention
Regulatory Engagement and Expedited Review Pathways: Engage regulators early for faster approvals.Utilize digital communication tools to engage regulatory agencies early in the drug development process, enabling faster feedback and exploration of expedited review pathways for accelerated approvals
Companion Diagnostics Development: Develop diagnostics for targeted recruitment and personalized treatment.Implement bioinformatics and genomics technologies to develop companion diagnostics that can identify patient subpopulations likely to benefit from the drug, aiding in targeted recruitment and personalized treatment
Data Standardization and Interoperability: Ensure seamless data exchange among research sites.Utilize interoperable electronic health record systems and health data standards to ensure seamless data exchange among different research sites, promoting efficient data aggregation and analysis
Use of AI and Predictive Analytics: Apply AI for drug candidate identification and data analysis.Leverage AI algorithms and predictive analytics to analyze large datasets, identify potential drug candidates, optimize trial designs, and predict treatment outcomes, accelerating the drug development process
R&D Investments: Improve the drug or expand indicationsUtilize computational modelling and simulation techniques to accelerate drug discovery and optimize drug development processes