Skip to main content

Hypothetical Document Embeddings

If we’re working with a similarity search-based index, like a vector store, then searching on raw questions may not work well because their embeddings may not be very similar to those of the relevant documents. Instead it might help to have the model generate a hypothetical relevant document, and then use that to perform similarity search. This is the key idea behind Hypothetical Document Embedding, or HyDE.

Let’s take a look at how we might perform search via hypothetical documents for our Q&A bot over the LangChain YouTube videos.

Setup​

Install dependencies​

tip

See this section for general instructions on installing integration packages.

yarn add @langchain/core zod

Set environment variables​

# Optional, use LangSmith for best-in-class observability
LANGSMITH_API_KEY=your-api-key
LANGCHAIN_TRACING_V2=true

Hypothetical document generation​

Ultimately generating a relevant hypothetical document reduces to trying to answer the user question. Since we’re desiging a Q&A bot for LangChain YouTube videos, we’ll provide some basic context about LangChain and prompt the model to use a more pedantic style so that we get more realistic hypothetical documents:

Pick your chat model:

Install dependencies

tip

See this section for general instructions on installing integration packages.

yarn add @langchain/openai

Add environment variables

OPENAI_API_KEY=your-api-key

Instantiate the model

import { ChatOpenAI } from "@langchain/openai";

const llm = new ChatOpenAI({
  model: "gpt-3.5-turbo-0125",
  temperature: 0
});
import { StringOutputParser } from "@langchain/core/output_parsers";
import { ChatPromptTemplate } from "@langchain/core/prompts";

const system = `You are an expert about a set of software for building LLM-powered applications called LangChain, LangGraph, LangServe, and LangSmith.

LangChain is a Python framework that provides a large set of integrations that can easily be composed to build LLM applications.
LangGraph is a Python package built on top of LangChain that makes it easy to build stateful, multi-actor LLM applications.
LangServe is a Python package built on top of LangChain that makes it easy to deploy a LangChain application as a REST API.
LangSmith is a platform that makes it easy to trace and test LLM applications.

Answer the user question as best you can. Answer as though you were writing a tutorial that addressed the user question. `;
const prompt = ChatPromptTemplate.fromMessages([
  ["system", system],
  ["human", "{question}"],
]);
const qaNoContext = prompt.pipe(llm).pipe(new StringOutputParser());
const answer = await qaNoContext.invoke({
  question:
    "how to use multi-modal models in a chain and turn chain into a rest api",
});
console.log(answer);
To use multi-modal models in a chain and turn the chain into a REST API, you can leverage the capabilities of LangChain, LangGraph, and LangServe. Here's a step-by-step guide on how to achieve this:

1. **Set up LangChain**: Start by installing LangChain, LangGraph, and LangServe in your Python environment. You can do this using pip:

```bash
pip install langchain langgraph langserve
```

2. **Build a Multi-Modal Model**: Create your multi-modal model using LangChain. LangChain provides integrations with various deep learning frameworks like TensorFlow, PyTorch, and Hugging Face Transformers. You can easily compose different modalities (text, image, audio, etc.) in your model.

3. **Use LangGraph for Stateful Multi-Actor Applications**: If your multi-modal model requires stateful interactions between different actors, you can use LangGraph to build such applications. LangGraph simplifies the process of managing state and interactions in your LLM application.

4. **Deploy as a REST API using LangServe**: Once you have built your multi-modal model and defined the interactions using LangGraph, you can deploy your chain as a REST API using LangServe. LangServe makes it easy to expose your LangChain application as a web service, allowing users to interact with your model through HTTP requests.

5. **Define Endpoints**: In your LangServe application, define the endpoints that correspond to different functionalities of your multi-modal model. For example, you can have endpoints for text input, image input, audio input, etc.

6. **Handle Requests**: Implement the logic to handle incoming requests in your LangServe application. Parse the input data, pass it through your multi-modal model, and return the results in the desired format.

7. **Start the LangServe Server**: Once you have defined your endpoints and request handling logic, start the LangServe server to make your multi-modal model accessible as a REST API. You can specify the host, port, and other configurations when starting the server.

By following these steps, you can effectively use multi-modal models in a chain and expose it as a REST API using LangChain, LangGraph, and LangServe. This approach allows you to build complex LLM applications with stateful interactions and make them accessible to users through a web interface.

Returning the hypothetical document and original question​

To increase our recall we may want to retrieve documents based on both the hypothetical document and the original question. We can easily return both like so:

import { RunnablePassthrough } from "@langchain/core/runnables";

const hydeChain = RunnablePassthrough.assign({
  hypotheticalDocument: qaNoContext,
});

await hydeChain.invoke({
  question:
    "how to use multi-modal models in a chain and turn chain into a rest api",
});
{
  question: "how to use multi-modal models in a chain and turn chain into a rest api",
  hypotheticalDocument: "To use multi-modal models in a chain and turn the chain into a REST API, you can leverage the capabi"... 1920 more characters
}

Using function-calling to get structured output​

If we were composing this technique with other query analysis techniques, we’d likely be using function calling to get out structured query objects. We can use function-calling for HyDE like so:

import { z } from "zod";

const querySchema = z.object({
  answer: z
    .string()
    .describe(
      "Answer the user question as best you can. Answer as though you were writing a tutorial that addressed the user question."
    ),
});

const system = `You are an expert about a set of software for building LLM-powered applications called LangChain, LangGraph, LangServe, and LangSmith.

LangChain is a Python framework that provides a large set of integrations that can easily be composed to build LLM applications.
LangGraph is a Python package built on top of LangChain that makes it easy to build stateful, multi-actor LLM applications.
LangServe is a Python package built on top of LangChain that makes it easy to deploy a LangChain application as a REST API.
LangSmith is a platform that makes it easy to trace and test LLM applications.`;

const prompt = ChatPromptTemplate.fromMessages([
  ["system", system],
  ["human", "{question}"],
]);
const llmWithTools = llm.withStructuredOutput(querySchema, {
  name: "Query",
});
const hydeChain = prompt.pipe(llmWithTools);
await hydeChain.invoke({
  question:
    "how to use multi-modal models in a chain and turn chain into a rest api",
});
{
  answer: "To use multi-modal models in a chain and turn the chain into a REST API, you can follow these steps:"... 713 more characters
}

Help us out by providing feedback on this documentation page: