Klex Git utilities

With the help of these tools, you can create a Git repository that relies on Klex for data processing and AI services.

Example use case and tutorial

Create a new Git repository.
```
mkdir goodevil
cd goodevil
git init
```

Create a dataset containing a book split into paragraphs.

mkdir datasets
cd datasets
wget https://gutenberg.org/cache/epub/4363/pg4363.txt -O book.txt
mkdir paragraphs
for i in {1..296}
do
awk `/^$i\. /{f=1} f; /^$((i+1))\. / && f{exit}' book.txt > paragraphs/$i.txt
done
rm book.txt
cd ..

At this point, you should have a directory called data/paragraphs/ with 296 file in it, each containing a paragraph from the book. See 256.txt for an example of an edge case.

Create a JavaScript function that generates LLM prompts.

mkdir functions
cat <<EOF > functions/tag.js
function(txt) {
return "Produce a list of useful search words for the following" +
  " paragraph of text. Present the list in lower case, separated by" +
  " commas, prefixed with the heading \"Tags: \", and ending in a" +
  " period. For example, if the paragraph talks primarily about" +
  " the way teachers view their profession, output this line:\n" +
  " Tags: teachers, teaching, profession.\n\n" +
  " Paragraphs: " + txt.replace(/^\d+[.] +/, "");
}
EOF

Create a JavaScript pipeline that tags each paragraph.

mkdir pipelines
cat <<EOF > pipelines/tag.js
function(api) {
return api.dataset("paragraphs")
  .map("tag")
  .map("llama3")
  .name("tagged");
}
EOF

Create a Klex config file.

cat <<EOF > klex.json
{
"project_name": "goodevil",
"datasets_dir": "datasets",
"functions_dir": "functions",
"pipelines_dir": "pipelines",
"klex_url": "https://dr.oscarkilo.com/klex",
"api_key_file": "klex.api_key"
}
EOF

Get an API key here and save it to a file.

echo "YOUR_API_KEY" > klex.api_key
echo "klex.api_key" > .gitignore

Add Klex Git utilities and hooks to your repository. First, clone this repository to the ../klex-git directory:

cd ..
git clone https://code.oscarkilo.com/klex-git
cd goodevil

Then,

cat <<EOF > go.mod
module goodevil
go 1.21
toolchain go1.21.3
replace oscarkilo.com/klex-git => ../klex-git
EOF
go get oscarkilo.com/klex-git
go run oscarkilo.com/klex-git/install

Commit your changes.
```
git add .
git commit -m "First!"
```

code.oscarkilo.com/klex-git/README.md

Klex Git utilities

Example use case and tutorial