A text embedding is a compressed, abstract representation of text data where text of arbitrary length can be represented as a vector of numbers. Think of them as a universal encoding for text, where similar items are close to each other while dissimilar items are farther apart. https://eugeneyan.com/writing/llm-patterns/
We have an API already to turn documents into text and we have an API to convert text into embeddings. Let's combine those together and store the results in Postgres.
Models have a limit on the prompt size. So it makes sense to split the documents into 512 character chunks and generate embeddings based on those chunks.
So in the database a large document will be split into many smaller entries in the database.
http://llm-api:8080/v1/embeddingsis 384 which has to match the settings in the DB