code.oscarkilo.com/klex-git

git clone https://code.oscarkilo.com/klex-git

Klex Git utilities

With the help of these tools, you can create a Git repository that relies on Klex for data processing and AI services.

Example use case and tutorial

1. Create a new Git repository.

   mkdir goodevil
   cd goodevil
   git init
   

2. Create a dataset containing a book split into paragraphs.

   mkdir datasets
   cd datasets
   wget https://gutenberg.org/cache/epub/4363/pg4363.txt -O book.txt
   mkdir paragraphs
   for i in {1..296}
   do
   awk `/^$i\. /{f=1} f; /^$((i+1))\. / && f{exit}' book.txt > paragraphs/$i.txt
   done
   rm book.txt
   cd ..
   

   At this point, you should have a directory called data/paragraphs/ with 296 file in it, each containing a paragraph from the book. See 256.txt for an example of an edge case.

3. Create a JavaScript function that generates LLM prompts.

   mkdir functions
   cat <<EOF > functions/tag.js
   function(txt) {
   return "Produce a list of useful search words for the following" +
     " paragraph of text. Present the list in lower case, separated by" +
     " commas, prefixed with the heading \"Tags: \", and ending in a" +
     " period. For example, if the paragraph talks primarily about" +
     " the way teachers view their profession, output this line:\n" +
     " Tags: teachers, teaching, profession.\n\n" +
     " Paragraphs: " + txt.replace(/^\d+[.] +/, "");
   }
   EOF
   

4. Create a JavaScript pipeline that tags each paragraph.

   mkdir pipelines
   cat <<EOF > pipelines/tag.js
   function(api) {
   return api.dataset("paragraphs")
     .map("tag")
     .map("llama3")
     .name("tagged");
   }
   EOF
   

5. Create a Klex config file.

   cat <<EOF > klex.json
   {
   "project_name": "goodevil",
   "datasets_dir": "datasets",
   "functions_dir": "functions",
   "pipelines_dir": "pipelines",
   "klex_url": "https://dr.oscarkilo.com/klex",
   "api_key_file": "klex.api_key"
   }
   EOF
   

6. Get an API key here and save it to a file.

   echo "YOUR_API_KEY" > klex.api_key
   echo "klex.api_key" > .gitignore
   

7. Add Klex Git utilities and hooks to your repository. First, clone this repository to the ../klex-git directory:

   cd ..
   git clone https://code.oscarkilo.com/klex-git
   cd goodevil
   

   Then,

   cat <<EOF > go.mod
   module goodevil
   go 1.21
   toolchain go1.21.3
   replace oscarkilo.com/klex-git => ../klex-git
   EOF
   go get oscarkilo.com/klex-git
   go run oscarkilo.com/klex-git/install
   

8. Commit your changes.

   git add .
   git commit -m "First!"