Commit edff69eef5
Unsigned
Layout: unified · split
content/blog/2024-01-13-local-llm.md added +94
| @@ -0,0 +1,94 @@ | |||
| 1 | +++ | ||
| 2 | date = 2024-01-13 | ||
| 3 | title = "Running Local LLMs on macOS and iOS" | ||
| 4 | description = "Finding some useful applications for running local LLMs on macOS and iOS." | ||
| 5 | +++ | ||
| 6 | |||
| 7 | ## Requirements | ||
| 8 | |||
| 9 | I've recently started playing with large language models (LLMs), mostly in the | ||
| 10 | popular chatbot form, as part of my job and have decided to see if there's a | ||
| 11 | consistent and reliable way to interact with these models on Apple devices | ||
| 12 | without sacrificing privacy or requiring in-depth technical setup. | ||
| 13 | |||
| 14 | My requirements for this test: | ||
| 15 | |||
| 16 | - Open source platform | ||
| 17 | - On-device model files | ||
| 18 | - Minimal required configuration | ||
| 19 | - Preferably pre-built, but a simple build process is acceptable | ||
| 20 | |||
| 21 | I tested a handful of apps and have summarized my favorite (so far) for macOS | ||
| 22 | and iOS below. | ||
| 23 | |||
| 24 | > TL;DR - Here are the two that met my requirements and I have found the easiest | ||
| 25 | > to install and use so far: | ||
| 26 | |||
| 27 | - macOS: [Ollama](https://ollama.ai/) | ||
| 28 | - iOS : [LLM Farm](https://llmfarm.site/) | ||
| 29 | |||
| 30 | ## macOS | ||
| 31 | |||
| 32 | [Ollama](https://ollama.ai/) is a simple Go application for macOS and Linux that | ||
| 33 | can run various LLMs locally. | ||
| 34 | |||
| 35 | For macOS, you can download the pplication on the [Ollama download | ||
| 36 | page](https://ollama.ai/download/mac) and install it by unzipping the | ||
| 37 | `Ollama.app` file and moving it to the `Applications` folder. | ||
| 38 | |||
| 39 | If you prefer the command line, you can run these commands after the download | ||
| 40 | finished: | ||
| 41 | |||
| 42 | ```sh | ||
| 43 | cd ~/Downloads && \ | ||
| 44 | unzip Ollama-darwin.zip && \ | ||
| 45 | mv ~/Downloads/Ollama.app /Applications/ | ||
| 46 | ``` | ||
| 47 | |||
| 48 | After running the app, the app will ask you to open a terminal and run the | ||
| 49 | default `llama2` model, which will open an interactive chat session in the | ||
| 50 | terminal. You can startfully using the application at this point. | ||
| 51 | |||
| 52 |  | ||
| 53 | |||
| 54 | If you don't want to use the default `llama2` model, you can download and run | ||
| 55 | additional models found on the [Models](https://ollama.ai/library) page. | ||
| 56 | |||
| 57 | To see the information for the currently-used model, you can run the `/show | ||
| 58 | info` command in the chat. | ||
| 59 | |||
| 60 | ![Model Info](https://img.cleberg.net/blog/20240113-local-llm/ollama_info.png | ||
| 61 | !"Model Info") | ||
| 62 | |||
| 63 | ## iOS | ||
| 64 | |||
| 65 | While there are a handful of decent macOS options, it was quite difficult to | ||
| 66 | find an iOS app that offered an open source platform without an extensive | ||
| 67 | configuration and building process. I found LLM Farm to be decent enough in | ||
| 68 | quality to sit at the top of my list - however, it's definitely not user | ||
| 69 | friendly enough for me to consider using it on a daily basis. | ||
| 70 | |||
| 71 | [LLM Farm](https://llmfarm.site/) is available on TestFlight, so there's no | ||
| 72 | manual build process required. However, you can view the [LLMFarm | ||
| 73 | repository](https://github.com/guinmoon/LLMFarm) if you wish. | ||
| 74 | |||
| 75 | The caveat is that you will have to manually download the model files from the | ||
| 76 | links in the | ||
| 77 | [models.md](https://github.com/guinmoon/LLMFarm/blob/main/models.md) file to | ||
| 78 | your iPhone to use the app - there's currently no option in the app to reach out | ||
| 79 | and grab the latest version of any supported model. | ||
| 80 | |||
| 81 | Once you have a file downloaded, you simply create a new chat and select the | ||
| 82 | downloaded model file and ensure the inference matches the requirement in the | ||
| 83 | `models.md` file. | ||
| 84 | |||
| 85 | See below for a test of the ORCA Mini v3 model: | ||
| 86 | |||
| 87 | | Chat List | Chat | | ||
| 88 | |--------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------| | ||
| 89 | |  |  | ||
| 90 | |||
| 91 | [Enchanted](https://github.com/AugustDev/enchanted) is also an iOS for private | ||
| 92 | AI models, but it requires a public-facing Ollama API, which did not meet my | ||
| 93 | "on device requirement." Nonetheless, it's an interesting looking app and I will | ||
| 94 | likely set it up to test soon. | ||