cmc/cleberg.net

 1#+date:        [2024-01-13 Sat 00:00:00]
 2#+title:       Local LLM: Run the Model, Keep the Data
 3#+description: Running local language models on macOS and iOS with open-source tools.
 4#+slug:        local-llm
 5#+filetags:    :linux:privacy:
 6
 7* Requirements
 8
 9I've recently started playing with large language models (LLMs), mostly in the
10popular chatbot form, as part of my job and have decided to see if there's a
11consistent and reliable way to interact with these models on Apple devices
12without sacrificing privacy or requiring in-depth technical setup.
13
14My requirements for this test:
15
16- Open source platform
17- On-device model files
18- Minimal required configuration
19- Preferably pre-built, but a simple build process is acceptable
20
21I tested a handful of apps and have summarized my favorite (so far) for macOS
22and iOS below.
23
24#+begin_quote
25TL;DR - Here are the two that met my requirements and I have found the easiest
26to install and use so far:
27#+end_quote
28
29- macOS: [[https://ollama.ai/][Ollama]]
30- iOS : [[https://llmfarm.site/][LLM Farm]]
31
32* macOS
33
34[[https://ollama.ai/][Ollama]] is a simple Go application for macOS and Linux that can run various LLMs
35locally.
36
37For macOS, you can download the application on the [[https://ollama.ai/download/mac][Ollama download page]] and
38install it by unzipping the =Ollama.app= file and moving it to the
39=Applications= folder.
40
41If you prefer the command line, you can run these commands after the
42download finished:
43
44#+begin_src sh
45cd ~/Downloads && \
46unzip Ollama-darwin.zip && \
47mv ~/Downloads/Ollama.app /Applications/
48#+end_src
49
50After running the app, the app will ask you to open a terminal and run the
51default =llama2= model, which will open an interactive chat session in the
52terminal. You can start using the application at this point.
53
54If you don't want to use the default =llama2= model, you can download and run
55additional models found on the [[https://ollama.ai/library][Models]] page.
56
57To see the information for the currently-used model, you can run the
58=/show info= command in the chat.
59
60** Community Integrations
61
62I highly recommend browsing the [[https://github.com/jmorganca/ollama#community-integrations][Community Integrations]] section of the project to
63see how you would prefer to extend Ollama beyond a simple command-line
64interface. There are options for APIs, browser UIs, advanced terminal
65configurations, and more.
66
67* iOS
68
69While there are a handful of decent macOS options, it was quite difficult to
70find an iOS app that offered an open source platform without an extensive
71configuration and building process. I found LLM Farm to be decent enough in
72quality to sit at the top of my list - however, it's definitely not user
73friendly enough for me to consider using it on a daily basis.
74
75[[https://llmfarm.site/][LLM Farm]] is available on TestFlight, so there's no manual build process
76required. However, you can view the [[https://github.com/guinmoon/LLMFarm][LLMFarm repository]] if you wish.
77
78The caveat is that you will have to manually download the model files from the
79links in the [[https://github.com/guinmoon/LLMFarm/blob/main/models.md][models.md]] file to your iPhone to use the app - there's currently no
80option in the app to reach out and grab the latest version of any supported
81model.
82
83Once you have a file downloaded, you simply create a new chat and select the
84downloaded model file and ensure the inference matches the requirement in the
85=models.md= file.
86
87[[https://github.com/AugustDev/enchanted][Enchanted]] is also an iOS app for private AI models, but it requires a public-facing
88Ollama API, which did not meet my "on device requirement." Nonetheless, it's an
89interesting looking app and I will likely set it up to test soon.