cmc/cleberg.net

  1#+date:        [2024-10-31 Thu 11:01:05]
  2#+title:       Ollama Code Assistant in VS Codium
  3#+description: Setting up Ollama as a local code assistant in VS Codium.
  4#+slug:        continue-ollama-code-assistant
  5#+filetags:    :linux:
  6
  7* Background
  8
  9As someone who doesn't do software development full time and intends to actually
 10enjoy it as a hobby, I've been interested in the /concept/ of code assistants
 11for a while. However, I had a few issues:
 12
 131. I'm not a full-time developer, so I wanted to actually learn all the various
 14   features and rules of the projects I was building.
 152. The quality was quite low until recently.
 163. I wanted to use an open-source solution. Ideally, one with a free tier so
 17   that I could test it before paying for a subscription.
 18
 19I recently discovered [[https://www.continue.dev/][Continue]], which allows for local LLM configurations via
 20[[https://ollama.com/][Ollama]], so I've decided to try it out and see how helpful a code assistant can
 21really be for someone like me, who doesn't do this professionally.
 22
 23* Installation
 24
 25Installation is a quick and painless process if you already have Ollama
 26installed, but I'll assume that we're starting from scratch.
 27
 28** Ollama
 29
 30First, install Ollama for your system. For macOS devices, I'd recommend using
 31Homebrew via the following command. However, you can also visit their website
 32and install the software manually.
 33
 34#+begin_src sh
 35brew install ollama
 36brew services start ollama
 37#+end_src
 38
 39Next, we will need to install the models requested by the Continue extension. By
 40default, Continue asks you to install the =llama3.1:8b= and =starcoder2:3b=
 41models. However, you can customize the configuration to specify different
 42models, if preferred.
 43
 44#+begin_src sh
 45ollama pull llama3.1:8b 
 46ollama pull starcoder2:3b
 47#+end_src
 48
 49At this point, Ollama should be running on your local machine and have two
 50models available for use.
 51
 52You can test this by visiting =http://localhost:11434/= in your browser or use
 53cURL in your shell:
 54
 55#+begin_src sh
 56curl http://localhost:11434
 57
 58# If running, the response will look like this:
 59# Ollama is running
 60#+end_src
 61
 62** Continue
 63
 64To get started with using Ollama as a code assistant in VS Codium, you'll first
 65need to install the Continue extension. This extension is available for free on
 66the Visual Studio Code marketplace and can be easily installed via the built-in
 67Extension view.
 68
 69Follow these steps:
 701. Open VS Codium and navigate to the Extensions view by clicking on the square
 71   icon next to the file menu or pressing =Ctrl+Shift+X= (Windows/Linux) or
 72   =Cmd+Shift+X= (Mac).
 732. Search for "Continue" in the Extensions marketplace.
 743. Click on the Continue extension and click Install.
 75
 76Once installed, restart VS Codium to ensure that the extension is properly
 77loaded.
 78
 79* Continue: Extension Configuration
 80
 81When you first install Continue, it will ask which service you'll be using. If
 82you select Ollama, it will check to make sure Ollama is running and you have
 83installed the two default models from Ollama.
 84
 85After this initial setup, you can open the =config.json= file by pressing =Cmd +
 86p= and entering => Continue: Open config.json=. This will show a config such as
 87the following:
 88
 89#+begin_src json
 90{
 91  "models": [
 92    {
 93      "title": "Llama 3.1 8B",
 94      "provider": "ollama",
 95      "model": "llama3.1:8b"
 96    },
 97    {
 98      "model": "claude-3-5-sonnet-20240620",
 99      "provider": "anthropic",
100      "apiKey": "",
101      "title": "Claude 3.5 Sonnet"
102    }
103  ],
104  "tabAutocompleteModel": {
105    "title": "Starcoder 3b",
106    "provider": "ollama",
107    "model": "starcoder2:3b"
108  },
109  "customCommands": [
110    {
111      "name": "test",
112      "prompt": "{{{ input }}}\n\nWrite a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
113      "description": "Write unit tests for highlighted code"
114    }
115  ],
116  "contextProviders": [
117    {
118      "name": "code",
119      "params": {}
120    },
121    {
122      "name": "docs",
123      "params": {}
124    },
125    {
126      "name": "diff",
127      "params": {}
128    },
129    {
130      "name": "terminal",
131      "params": {}
132    },
133    {
134      "name": "problems",
135      "params": {}
136    },
137    {
138      "name": "folder",
139      "params": {}
140    },
141    {
142      "name": "codebase",
143      "params": {}
144    }
145  ],
146  "slashCommands": [
147    {
148      "name": "edit",
149      "description": "Edit selected code"
150    },
151    {
152      "name": "comment",
153      "description": "Write comments for the selected code"
154    },
155    {
156      "name": "share",
157      "description": "Export the current chat session to markdown"
158    },
159    {
160      "name": "cmd",
161      "description": "Generate a shell command"
162    },
163    {
164      "name": "commit",
165      "description": "Generate a git commit message"
166    }
167  ],
168  "embeddingsProvider": {
169    "provider": "ollama",
170    "model": "nomic-embed-text"
171  }
172}
173#+end_src
174
175You can modify this file with many different customizations. Refer to the
176[[https://docs.continue.dev/customize/config][Configuration options]] page for more information.
177
178* Use Cases
179
180While I'm sure there are a ton of use cases that I can't think of, I decided to
181test it out with this blog and some basic Python scripts I am currently using.
182Here are the most common ones I've used so far:
183
184- Improving README documentation
185- Refactor my =salary_visualization.py= script to align with PEP8.
186- Auto-complete thoughts and suggest further ideas for topics in this blog post.
187
188As an example, the following list of possible use cases was auto-generated by
189Continue:
190
191- Auto-complete function names and variables: With Ollama enabled, typing a few
192  characters into the editor will suggest matching functions or variables from
193  the entire project.
194- Code suggestions for common tasks: Ollama can provide suggestions for common
195  programming tasks, such as converting types or formatting code.
196- Live coding assistance: As you type, Ollama can offer live suggestions and
197  corrections to help ensure your code is correct.
198
199** Screenshots
200
201Below are a few screenshots from my current VS Codium window:
202
203#+caption: Available Continue Commands
204#+attr_html: :alt List of commands available from the Continue package.
205[[https://img.cleberg.net/blog/20241031-continue-ollama-code-assistant/continue_commands.webp]]
206
207#+caption: Continue Fullscreen Chat Window
208#+attr_html: :alt Continue chat window in full screen mode.
209[[https://img.cleberg.net/blog/20241031-continue-ollama-code-assistant/continue_fullscreen.webp]]
210
211#+caption: Inline Hotkeys
212#+attr_html: :alt View of the Add to chat (Cmd+L) and Edit highlighted code (Cmd+I) options when typing inline.
213[[https://img.cleberg.net/blog/20241031-continue-ollama-code-assistant/continue_inline_hotkeys.webp]]
214
215#+caption: Inline Editing Suggestions
216#+attr_html: :alt Highlighted red and green sections showing suggestions from the agent.
217[[https://img.cleberg.net/blog/20241031-continue-ollama-code-assistant/continue_inline.webp]]
218
219#+caption: Sidebar Context Window
220#+attr_html: :alt View of the sidebar context window open side-by-side with an edited file.
221[[https://img.cleberg.net/blog/20241031-continue-ollama-code-assistant/continue_sidebar.webp]]
222
223* Conclusion
224
225As it stands, it seems that the current iteration of code completion and review
226models from Ollama are quite good for my use case. In particular, it is able to
227suggest logical continuations of my thoughts in a blog post, generate accurate
228documentation based on my files, explain code to me with references within the
229project, and align my existing files to standards.
230
231However, it is not perfect. I have noticed that it often goes off in a random
232direction, unrelated to the intent of what I'm writing (either blogging or
233programming). It also struggles to understand the full context without clear,
234specific, repeated instructions to refer to certain files, standards, etc. while
235suggesting improvements.
236
237All together, I think it's useful enough to suggest as an add-on tool, but I
238would be highly skeptical of any suggestions it provides.