Commit edff69eef5
Unsigned
Layout: unified · split
content/blog/2024-01-13-local-llm.md added +94
| @@ -0,0 +1,94 @@ | ||
| 1 | +++ | |
| 2 | date = 2024-01-13 | |
| 3 | title = "Running Local LLMs on macOS and iOS" | |
| 4 | description = "Finding some useful applications for running local LLMs on macOS and iOS." | |
| 5 | +++ | |
| 6 | ||
| 7 | ## Requirements | |
| 8 | ||
| 9 | I've recently started playing with large language models (LLMs), mostly in the | |
| 10 | popular chatbot form, as part of my job and have decided to see if there's a | |
| 11 | consistent and reliable way to interact with these models on Apple devices | |
| 12 | without sacrificing privacy or requiring in-depth technical setup. | |
| 13 | ||
| 14 | My requirements for this test: | |
| 15 | ||
| 16 | - Open source platform | |
| 17 | - On-device model files | |
| 18 | - Minimal required configuration | |
| 19 | - Preferably pre-built, but a simple build process is acceptable | |
| 20 | ||
| 21 | I tested a handful of apps and have summarized my favorite (so far) for macOS | |
| 22 | and iOS below. | |
| 23 | ||
| 24 | > TL;DR - Here are the two that met my requirements and I have found the easiest | |
| 25 | > to install and use so far: | |
| 26 | ||
| 27 | - macOS: [Ollama](https://ollama.ai/) | |
| 28 | - iOS : [LLM Farm](https://llmfarm.site/) | |
| 29 | ||
| 30 | ## macOS | |
| 31 | ||
| 32 | [Ollama](https://ollama.ai/) is a simple Go application for macOS and Linux that | |
| 33 | can run various LLMs locally. | |
| 34 | ||
| 35 | For macOS, you can download the pplication on the [Ollama download | |
| 36 | page](https://ollama.ai/download/mac) and install it by unzipping the | |
| 37 | `Ollama.app` file and moving it to the `Applications` folder. | |
| 38 | ||
| 39 | If you prefer the command line, you can run these commands after the download | |
| 40 | finished: | |
| 41 | ||
| 42 | ```sh | |
| 43 | cd ~/Downloads && \ | |
| 44 | unzip Ollama-darwin.zip && \ | |
| 45 | mv ~/Downloads/Ollama.app /Applications/ | |
| 46 | ``` | |
| 47 | ||
| 48 | After running the app, the app will ask you to open a terminal and run the | |
| 49 | default `llama2` model, which will open an interactive chat session in the | |
| 50 | terminal. You can startfully using the application at this point. | |
| 51 | ||
| 52 |  | |
| 53 | ||
| 54 | If you don't want to use the default `llama2` model, you can download and run | |
| 55 | additional models found on the [Models](https://ollama.ai/library) page. | |
| 56 | ||
| 57 | To see the information for the currently-used model, you can run the `/show | |
| 58 | info` command in the chat. | |
| 59 | ||
| 60 | ![Model Info](https://img.cleberg.net/blog/20240113-local-llm/ollama_info.png | |
| 61 | !"Model Info") | |
| 62 | ||
| 63 | ## iOS | |
| 64 | ||
| 65 | While there are a handful of decent macOS options, it was quite difficult to | |
| 66 | find an iOS app that offered an open source platform without an extensive | |
| 67 | configuration and building process. I found LLM Farm to be decent enough in | |
| 68 | quality to sit at the top of my list - however, it's definitely not user | |
| 69 | friendly enough for me to consider using it on a daily basis. | |
| 70 | ||
| 71 | [LLM Farm](https://llmfarm.site/) is available on TestFlight, so there's no | |
| 72 | manual build process required. However, you can view the [LLMFarm | |
| 73 | repository](https://github.com/guinmoon/LLMFarm) if you wish. | |
| 74 | ||
| 75 | The caveat is that you will have to manually download the model files from the | |
| 76 | links in the | |
| 77 | [models.md](https://github.com/guinmoon/LLMFarm/blob/main/models.md) file to | |
| 78 | your iPhone to use the app - there's currently no option in the app to reach out | |
| 79 | and grab the latest version of any supported model. | |
| 80 | ||
| 81 | Once you have a file downloaded, you simply create a new chat and select the | |
| 82 | downloaded model file and ensure the inference matches the requirement in the | |
| 83 | `models.md` file. | |
| 84 | ||
| 85 | See below for a test of the ORCA Mini v3 model: | |
| 86 | ||
| 87 | | Chat List | Chat | | |
| 88 | |--------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------| | |
| 89 | |  |  | |
| 90 | ||
| 91 | [Enchanted](https://github.com/AugustDev/enchanted) is also an iOS for private | |
| 92 | AI models, but it requires a public-facing Ollama API, which did not meet my | |
| 93 | "on device requirement." Nonetheless, it's an interesting looking app and I will | |
| 94 | likely set it up to test soon. | |