Skip to main content

Citations

How can we get a model to cite which parts of the source documents it referenced in its response?

To explore some techniques for extracting citations, let’s first create a simple RAG chain. To start we’ll just retrieve from the web using the TavilySearchAPIRetriever.

Setup​

Dependencies​

We’ll use an OpenAI chat model and embeddings and a Memory vector store in this walkthrough, but everything shown here works with any ChatModel or LLM, Embeddings, and VectorStore or Retriever.

We’ll use the following packages:

npm install --save langchain @langchain/community @langchain/openai

We need to set environment variables for Tavily Search & OpenAI:

export OPENAI_API_KEY=YOUR_KEY
export TAVILY_API_KEY=YOUR_KEY

LangSmith​

Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls. As these applications get more and more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent. The best way to do this is with LangSmith.

Note that LangSmith is not needed, but it is helpful. If you do want to use LangSmith, after you sign up at the link above, make sure to set your environment variables to start logging traces:

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=YOUR_KEY

Initial setup​

import { TavilySearchAPIRetriever } from "@langchain/community/retrievers/tavily_search_api";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatOpenAI } from "@langchain/openai";

const llm = new ChatOpenAI({
  model: "gpt-3.5-turbo",
  temperature: 0,
});
const retriever = new TavilySearchAPIRetriever({
  k: 6,
});
const prompt = ChatPromptTemplate.fromMessages([
  [
    "system",
    "You're a helpful AI assistant. Given a user question and some web article snippets, answer the user question. If none of the articles answer the question, just say you don't know.\n\nHere are the web articles:{context}",
  ],
  ["human", "{question}"],
]);

Now that we’ve got a model, retriever and prompt, let’s chain them all together. We’ll need to add some logic for formatting our retrieved Documents to a string that can be passed to our prompt. We’ll make it so our chain returns both the answer and the retrieved Documents.

import { Document } from "@langchain/core/documents";
import { StringOutputParser } from "@langchain/core/output_parsers";
import { RunnableMap, RunnablePassthrough } from "@langchain/core/runnables";

/**
 * Format the documents into a readable string.
 */
const formatDocs = (input: Record<string, any>): string => {
  const { docs } = input;
  return (
    "\n\n" +
    docs
      .map(
        (doc: Document) =>
          `Article title: ${doc.metadata.title}\nArticle Snippet: ${doc.pageContent}`
      )
      .join("\n\n")
  );
};
// subchain for generating an answer once we've done retrieval
const answerChain = prompt.pipe(llm).pipe(new StringOutputParser());
const map = RunnableMap.from({
  question: new RunnablePassthrough(),
  docs: retriever,
});
// complete chain that calls the retriever -> formats docs to string -> runs answer subchain -> returns just the answer and retrieved docs.
const chain = map
  .assign({ context: formatDocs })
  .assign({ answer: answerChain })
  .pick(["answer", "docs"]);
await chain.invoke("How fast are cheetahs?");
{
  answer: "Cheetahs are capable of reaching speeds as high as 75 mph or 120 km/h. Their average speed, however,"... 29 more characters,
  docs: [
    Document {
      pageContent: "Now, their only hope lies in the hands of human conservationists, working tirelessly to save the che"... 880 more characters,
      metadata: {
        title: "How Fast Are Cheetahs, and Other Fascinating Facts About the World's ...",
        source: "https://www.discovermagazine.com/planet-earth/how-fast-are-cheetahs-and-other-fascinating-facts-abou"... 21 more characters,
        score: 0.93715,
        images: null
      }
    },
    Document {
      pageContent: "If a lion comes along, the cheetah will abandon its catch -- it can't fight off a lion, and chances "... 911 more characters,
      metadata: {
        title: "What makes a cheetah run so fast? | HowStuffWorks",
        source: "https://animals.howstuffworks.com/mammals/cheetah-speed.htm",
        score: 0.93412,
        images: null
      }
    },
    Document {
      pageContent: "The science of cheetah speed\n" +
        "The cheetah (Acinonyx jubatus) is the fastest land animal on Earth, cap"... 738 more characters,
      metadata: {
        title: "How Fast Can a Cheetah Run? - ThoughtCo",
        source: "https://www.thoughtco.com/how-fast-can-a-cheetah-run-4587031",
        score: 0.93134,
        images: null
      }
    },
    Document {
      pageContent: "One of two videos from National Geographic's award-winning multimedia coverage of cheetahs in the ma"... 60 more characters,
      metadata: {
        title: "The Science of a Cheetah's Speed | National Geographic",
        source: "https://www.youtube.com/watch?v=icFMTB0Pi0g",
        score: 0.93109,
        images: null
      }
    },
    Document {
      pageContent: "Contact Us − +\n" +
        "Address\n" +
        "Smithsonian's National Zoo & Conservation Biology Institute  3001 Connecticut"... 1343 more characters,
      metadata: {
        title: "Cheetah | Smithsonian's National Zoo and Conservation Biology Institute",
        source: "https://nationalzoo.si.edu/animals/cheetah",
        score: 0.92938,
        images: null
      }
    },
    Document {
      pageContent: "Threats to the Cheetah’s Reign\n" +
        "As unparalleled as the cheetah’s speed might be, they face numerous c"... 907 more characters,
      metadata: {
        title: "How Fast Can a Cheetah Run? The Secrets Behind Its Incredible Speed",
        source: "https://www.explorationjunkie.com/how-fast-can-a-cheetah-run/",
        score: 0.871,
        images: null
      }
    }
  ]
}

LangSmith trace here

Function-calling​

Cite documents​

Let’s try using OpenAI function-calling to make the model specify which of the provided documents it’s actually referencing when answering. LangChain has some utils for converting objects or zod objects to the JSONSchema format expected by OpenAI, so we’ll use that to define our functions:

import { z } from "zod";
import { StructuredTool } from "@langchain/core/tools";
import { formatToOpenAITool } from "@langchain/openai";

class CitedAnswer extends StructuredTool {
  name = "cited_answer";

  description =
    "Answer the user question based only on the given sources, and cite the sources used.";

  schema = z.object({
    answer: z
      .string()
      .describe(
        "The answer to the user question, which is based only on the given sources."
      ),
    citations: z
      .array(z.number())
      .describe(
        "The integer IDs of the SPECIFIC sources which justify the answer."
      ),
  });

  constructor() {
    super();
  }

  _call(input: z.infer<(typeof this)["schema"]>): Promise<string> {
    return Promise.resolve(JSON.stringify(input, null, 2));
  }
}

const asOpenAITool = formatToOpenAITool(new CitedAnswer());
const tools1 = [asOpenAITool];

Let’s see what the model output is like when we pass in our functions and a user input:

const llmWithTool1 = llm.bind({
  tools: tools1,
  tool_choice: asOpenAITool,
});

const exampleQ = `What Brian's height?

Source: 1
Information: Suzy is 6'2"

Source: 2
Information: Jeremiah is blonde

Source: 3
Information: Brian is 3 inches shorted than Suzy`;

await llmWithTool1.invoke(exampleQ);
AIMessage {
  lc_serializable: true,
  lc_kwargs: {
    content: "",
    additional_kwargs: {
      function_call: undefined,
      tool_calls: [
        {
          id: "call_WzPoDCIRQ1pCah8k93cVrqex",
          type: "function",
          function: [Object]
        }
      ]
    }
  },
  lc_namespace: [ "langchain_core", "messages" ],
  content: "",
  name: undefined,
  additional_kwargs: {
    function_call: undefined,
    tool_calls: [
      {
        id: "call_WzPoDCIRQ1pCah8k93cVrqex",
        type: "function",
        function: {
          name: "cited_answer",
          arguments: "{\n" +
            `  "answer": "Brian's height is 6'2\\" - 3 inches",\n` +
            '  "citations": [1, 3]\n' +
            "}"
        }
      }
    ]
  }
}

LangSmith trace here

We’ll add an output parser to convert the OpenAI API response to a nice object. We use the JsonOutputKeyToolsParser for this:

import { JsonOutputKeyToolsParser } from "langchain/output_parsers";

const outputParser = new JsonOutputKeyToolsParser({
  keyName: "cited_answer",
  returnSingle: true,
});

await llmWithTool1.pipe(outputParser).invoke(exampleQ);
{ answer: `Brian's height is 6'2" - 3 inches`, citations: [ 1, 3 ] }

LangSmith trace here

Now we’re ready to put together our chain

import { Document } from "@langchain/core/documents";

const formatDocsWithId = (docs: Array<Document>): string => {
  return (
    "\n\n" +
    docs
      .map(
        (doc: Document, idx: number) =>
          `Source ID: ${idx}\nArticle title: ${doc.metadata.title}\nArticle Snippet: ${doc.pageContent}`
      )
      .join("\n\n")
  );
};
// subchain for generating an answer once we've done retrieval
const answerChain1 = prompt.pipe(llmWithTool1).pipe(outputParser);
const map1 = RunnableMap.from({
  question: new RunnablePassthrough(),
  docs: retriever,
});
// complete chain that calls the retriever -> formats docs to string -> runs answer subchain -> returns just the answer and retrieved docs.
const chain1 = map1
  .assign({
    context: (input: { docs: Array<Document> }) => formatDocsWithId(input.docs),
  })
  .assign({ cited_answer: answerChain1 })
  .pick(["cited_answer", "docs"]);
await chain1.invoke("How fast are cheetahs?");
{
  cited_answer: {
    answer: "Cheetahs can reach speeds of up to 75 mph (120 km/h).",
    citations: [ 3 ]
  },
  docs: [
    Document {
      pageContent: "The speeds attained by the cheetah may be only slightly greater than those achieved by the pronghorn"... 2527 more characters,
      metadata: {
        title: "Cheetah - Wikipedia",
        source: "https://en.wikipedia.org/wiki/Cheetah",
        score: 0.97773,
        images: null
      }
    },
    Document {
      pageContent: "Contact Us − +\n" +
        "Address\n" +
        "Smithsonian's National Zoo & Conservation Biology Institute  3001 Connecticut"... 1343 more characters,
      metadata: {
        title: "Cheetah | Smithsonian's National Zoo and Conservation Biology Institute",
        source: "https://nationalzoo.si.edu/animals/cheetah",
        score: 0.9681,
        images: null
      }
    },
    Document {
      pageContent: "The maximum speed cheetahs have been measured at is 114 km (71 miles) per hour, and they routinely r"... 1048 more characters,
      metadata: {
        title: "Cheetah | Description, Speed, Habitat, Diet, Cubs, & Facts",
        source: "https://www.britannica.com/animal/cheetah-mammal",
        score: 0.9459,
        images: null
      }
    },
    Document {
      pageContent: "The science of cheetah speed\n" +
        "The cheetah (Acinonyx jubatus) is the fastest land animal on Earth, cap"... 738 more characters,
      metadata: {
        title: "How Fast Can a Cheetah Run? - ThoughtCo",
        source: "https://www.thoughtco.com/how-fast-can-a-cheetah-run-4587031",
        score: 0.93957,
        images: null
      }
    },
    Document {
      pageContent: "One of two videos from National Geographic's award-winning multimedia coverage of cheetahs in the ma"... 60 more characters,
      metadata: {
        title: "The Science of a Cheetah's Speed | National Geographic",
        source: "https://www.youtube.com/watch?v=icFMTB0Pi0g",
        score: 0.92814,
        images: null
      }
    },
    Document {
      pageContent: "If a lion comes along, the cheetah will abandon its catch -- it can't fight off a lion, and chances "... 911 more characters,
      metadata: {
        title: "What makes a cheetah run so fast? | HowStuffWorks",
        source: "https://animals.howstuffworks.com/mammals/cheetah-speed.htm",
        score: 0.85762,
        images: null
      }
    }
  ]
}

LangSmith trace here

Cite snippets​

What if we want to cite actual text spans? We can try to get our model to return these, too.

Aside: Note that if we break up our documents so that we have many documents with only a sentence or two instead of a few long documents, citing documents becomes roughly equivalent to citing snippets, and may be easier for the model because the model just needs to return an identifier for each snippet instead of the actual text. Probably worth trying both approaches and evaluating.

const citationSchema = z.object({
  sourceId: z
    .number()
    .describe(
      "The integer ID of a SPECIFIC source which justifies the answer."
    ),
  quote: z
    .string()
    .describe(
      "The VERBATIM quote from the specified source that justifies the answer."
    ),
});

class QuotedAnswer extends StructuredTool {
  name = "quoted_answer";

  description =
    "Answer the user question based only on the given sources, and cite the sources used.";

  schema = z.object({
    answer: z
      .string()
      .describe(
        "The answer to the user question, which is based only on the given sources."
      ),
    citations: z
      .array(citationSchema)
      .describe("Citations from the given sources that justify the answer."),
  });

  constructor() {
    super();
  }

  _call(input: z.infer<(typeof this)["schema"]>): Promise<string> {
    return Promise.resolve(JSON.stringify(input, null, 2));
  }
}

const quotedAnswerTool = formatToOpenAITool(new QuotedAnswer());
const tools2 = [quotedAnswerTool];
import { Document } from "@langchain/core/documents";

const outputParser2 = new JsonOutputKeyToolsParser({
  keyName: "quoted_answer",
  returnSingle: true,
});
const llmWithTool2 = llm.bind({
  tools: tools2,
  tool_choice: quotedAnswerTool,
});
const answerChain2 = prompt.pipe(llmWithTool2).pipe(outputParser2);
const map2 = RunnableMap.from({
  question: new RunnablePassthrough(),
  docs: retriever,
});
// complete chain that calls the retriever -> formats docs to string -> runs answer subchain -> returns just the answer and retrieved docs.
const chain2 = map2
  .assign({
    context: (input: { docs: Array<Document> }) => formatDocsWithId(input.docs),
  })
  .assign({ quoted_answer: answerChain2 })
  .pick(["quoted_answer", "docs"]);
await chain2.invoke("How fast are cheetahs?");
{
  quoted_answer: {
    answer: "Cheetahs can reach speeds of up to 70 mph.",
    citations: [
      {
        sourceId: 0,
        quote: "We’ve mentioned that these guys can reach speeds of up to 70 mph"
      },
      {
        sourceId: 2,
        quote: "The maximum speed cheetahs have been measured at is 114 km (71 miles) per hour, and they routinely r"... 72 more characters
      },
      {
        sourceId: 5,
        quote: "Cheetahs—the fastest land mammals on the planet—are able to reach speeds of up to 70 mph"
      }
    ]
  },
  docs: [
    Document {
      pageContent: "They are surprisingly graceful\n" +
        "Cheetahs are very lithe-they move quickly and full-grown adults weigh"... 824 more characters,
      metadata: {
        title: "How Fast Are Cheetahs - Proud Animal",
        source: "https://www.proudanimal.com/2024/01/27/fast-cheetahs/",
        score: 0.97272,
        images: null
      }
    },
    Document {
      pageContent: "The Science of Speed\n" +
        "Instead, previous research has shown that the fastest animals are not the large"... 743 more characters,
      metadata: {
        title: "Now Scientists Can Accurately Guess The Speed Of Any Animal",
        source: "https://www.nationalgeographic.com/animals/article/Animal-speed-size-cheetahs",
        score: 0.96532,
        images: null
      }
    },
    Document {
      pageContent: "The maximum speed cheetahs have been measured at is 114 km (71 miles) per hour, and they routinely r"... 1048 more characters,
      metadata: {
        title: "Cheetah | Description, Speed, Habitat, Diet, Cubs, & Facts",
        source: "https://www.britannica.com/animal/cheetah-mammal",
        score: 0.95122,
        images: null
      }
    },
    Document {
      pageContent: "Now, their only hope lies in the hands of human conservationists, working tirelessly to save the che"... 880 more characters,
      metadata: {
        title: "How Fast Are Cheetahs, and Other Fascinating Facts About the World's ...",
        source: "https://www.discovermagazine.com/planet-earth/how-fast-are-cheetahs-and-other-fascinating-facts-abou"... 21 more characters,
        score: 0.92667,
        images: null
      }
    },
    Document {
      pageContent: "Contact Us − +\n" +
        "Address\n" +
        "Smithsonian's National Zoo & Conservation Biology Institute  3001 Connecticut"... 1343 more characters,
      metadata: {
        title: "Cheetah | Smithsonian's National Zoo and Conservation Biology Institute",
        source: "https://nationalzoo.si.edu/animals/cheetah",
        score: 0.91253,
        images: null
      }
    },
    Document {
      pageContent: "Cheetahs—the fastest land mammals on the planet—are incredible creatures. They're able to reach spee"... 95 more characters,
      metadata: {
        title: "Amazing Cheetah Facts | How Fast is a Cheetah? - Popular Mechanics",
        source: "https://www.popularmechanics.com/science/animals/g30021998/facts-about-cheetahs/",
        score: 0.87489,
        images: null
      }
    }
  ]
}

LangSmith trace here

Direct prompting​

Most models don’t yet support function-calling. We can achieve similar results with direct prompting. Let’s see what this looks like using an Anthropic chat model that is particularly proficient in working with XML:

Setup​

Install the LangChain Anthropic integration package:

npm install @langchain/anthropic

Add your Anthropic API key to your environment:

export ANTHROPIC_API_KEY=YOUR_KEY
import { ChatAnthropic } from "@langchain/anthropic";
import { ChatPromptTemplate } from "@langchain/core/prompts";

const anthropic = new ChatAnthropic({
  model: "claude-instant-1.2",
});
const system = `You're a helpful AI assistant. Given a user question and some web article snippets,
answer the user question and provide citations. If none of the articles answer the question, just say you don't know.

Remember, you must return both an answer and citations. A citation consists of a VERBATIM quote that
justifies the answer and the ID of the quote article. Return a citation for every quote across all articles
that justify the answer. Use the following format for your final output:

<cited_answer>
    <answer></answer>
    <citations>
        <citation><source_id></source_id><quote></quote></citation>
        <citation><source_id></source_id><quote></quote></citation>
        ...
    </citations>
</cited_answer>

Here are the web articles:{context}`;

const anthropicPrompt = ChatPromptTemplate.fromMessages([
  ["system", system],
  ["human", "{question}"],
]);
import { XMLOutputParser } from "@langchain/core/output_parsers";
import { Document } from "@langchain/core/documents";
import {
  RunnableLambda,
  RunnablePassthrough,
  RunnableMap,
} from "@langchain/core/runnables";

const formatDocsToXML = (docs: Array<Document>): string => {
  const formatted: Array<string> = [];
  docs.forEach((doc, idx) => {
    const docStr = `<source id="${idx}">
  <title>${doc.metadata.title}</title>
  <article_snippet>${doc.pageContent}</article_snippet>
</source>`;
    formatted.push(docStr);
  });
  return `\n\n<sources>${formatted.join("\n")}</sources>`;
};

const format3 = new RunnableLambda({
  func: (input: { docs: Array<Document> }) => formatDocsToXML(input.docs),
});
const answerChain = anthropicPrompt
  .pipe(anthropic)
  .pipe(new XMLOutputParser())
  .pipe(
    new RunnableLambda({
      func: (input: { cited_answer: any }) => input.cited_answer,
    })
  );
const map3 = RunnableMap.from({
  question: new RunnablePassthrough(),
  docs: retriever,
});
const chain3 = map3
  .assign({ context: format3 })
  .assign({ cited_answer: answerChain })
  .pick(["cited_answer", "docs"]);
await chain3.invoke("How fast are cheetahs?");
{
  cited_answer: [
    {
      answer: "Cheetahs can reach top speeds of between 60 to 70 mph."
    },
    {
      citations: [
        { citation: [Array] },
        { citation: [Array] },
        { citation: [Array] }
      ]
    }
  ],
  docs: [
    Document {
      pageContent: "A cheetah's muscular tail helps control their steering and keep their balance when running very fast"... 210 more characters,
      metadata: {
        title: "75 Amazing Cheetah Facts Your Kids Will Love (2024)",
        source: "https://www.mkewithkids.com/post/cheetah-facts-for-kids/",
        score: 0.97081,
        images: null
      }
    },
    Document {
      pageContent: "The maximum speed cheetahs have been measured at is 114 km (71 miles) per hour, and they routinely r"... 1048 more characters,
      metadata: {
        title: "Cheetah | Description, Speed, Habitat, Diet, Cubs, & Facts",
        source: "https://www.britannica.com/animal/cheetah-mammal",
        score: 0.96824,
        images: null
      }
    },
    Document {
      pageContent: "The Science of Speed\n" +
        "Instead, previous research has shown that the fastest animals are not the large"... 743 more characters,
      metadata: {
        title: "Now Scientists Can Accurately Guess The Speed Of Any Animal",
        source: "https://www.nationalgeographic.com/animals/article/Animal-speed-size-cheetahs",
        score: 0.96237,
        images: null
      }
    },
    Document {
      pageContent: "Contact Us − +\n" +
        "Address\n" +
        "Smithsonian's National Zoo & Conservation Biology Institute  3001 Connecticut"... 1343 more characters,
      metadata: {
        title: "Cheetah | Smithsonian's National Zoo and Conservation Biology Institute",
        source: "https://nationalzoo.si.edu/animals/cheetah",
        score: 0.94565,
        images: null
      }
    },
    Document {
      pageContent: "They are surprisingly graceful\n" +
        "Cheetahs are very lithe-they move quickly and full-grown adults weigh"... 824 more characters,
      metadata: {
        title: "How Fast Are Cheetahs - Proud Animal",
        source: "https://www.proudanimal.com/2024/01/27/fast-cheetahs/",
        score: 0.91795,
        images: null
      }
    },
    Document {
      pageContent: "Cheetahs are the world's fastest land animal. They can reach a speed of 69.5 miles per hour in just "... 100 more characters,
      metadata: {
        title: "How fast is Tyreek Hill? 'The Cheetah' lives up to 40 time, Next Gen ...",
        source: "https://www.sportingnews.com/us/nfl/news/fast-tyreek-hill-40-time-speed-chiefs/1cekgawhz39wr1tr472e4"... 5 more characters,
        score: 0.83505,
        images: null
      }
    }
  ]
}

LangSmith trace here

Retrieval post-processing​

Another approach is to post-process our retrieved documents to compress the content, so that the source content is already minimal enough that we don’t need the model to cite specific sources or spans. For example, we could break up each document into a sentence or two, embed those and keep only the most relevant ones. LangChain has some built-in components for this. Here we’ll use a RecursiveCharacterTextSplitter, which creates chunks of a specified size by splitting on separator substrings, and an EmbeddingsFilter, which keeps only the texts with the most relevant embeddings.

import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import { EmbeddingsFilter } from "langchain/retrievers/document_compressors/embeddings_filter";
import { OpenAIEmbeddings } from "@langchain/openai";
import { DocumentInterface } from "@langchain/core/documents";
import { RunnableMap, RunnablePassthrough } from "@langchain/core/runnables";

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 400,
  chunkOverlap: 0,
  separators: ["\n\n", "\n", ".", " "],
  keepSeparator: false,
});

const compressor = new EmbeddingsFilter({
  embeddings: new OpenAIEmbeddings(),
  k: 10,
});

const splitAndFilter = async (input): Promise<Array<DocumentInterface>> => {
  const { docs, question } = input;
  const splitDocs = await splitter.splitDocuments(docs);
  const statefulDocs = await compressor.compressDocuments(splitDocs, question);
  return statefulDocs;
};

const retrieveMap = RunnableMap.from({
  question: new RunnablePassthrough(),
  docs: retriever,
});

const retrieve = retrieveMap.pipe(splitAndFilter);
const docs = await retrieve.invoke("How fast are cheetahs?");
for (const doc of docs) {
  console.log(doc.pageContent, "\n\n");
}
The maximum speed cheetahs have been measured at is 114 km (71 miles) per hour, and they routinely reach velocities of 80–100 km (50–62 miles) per hour while pursuing prey.
cheetah,
(Acinonyx jubatus),


The science of cheetah speed
The cheetah (Acinonyx jubatus) is the fastest land animal on Earth, capable of reaching speeds as high as 75 mph or 120 km/h. Cheetahs are predators that sneak up on their prey and sprint a short distance to chase and attack.
 Key Takeaways: How Fast Can a Cheetah Run?
Fastest Cheetah on Earth


Built for speed, the cheetah can accelerate from zero to 45 in just 2.5 seconds and reach top speeds of 60 to 70 mph, making it the fastest land mammal! Fun Facts
Conservation Status
Cheetah News
Taxonomic Information
Animal News
NZCBI staff in Front Royal, Virginia, are mourning the loss of Walnut, a white-naped crane who became an internet sensation for choosing one of her keepers as her mate.


Scientists calculate a cheetah's top speed is 75 mph, but the fastest recorded speed is somewhat slower. The top 10 fastest animals are:


The pronghorn, an American animal resembling an antelope, is the fastest land animal in the Western Hemisphere. While a cheetah's top speed ranges from 65 to 75 mph (104 to 120 km/h), its average speed is only 40 mph (64 km/hr), punctuated by short bursts at its top speed. Basically, if a predator threatens to take a cheetah's kill or attack its young, a cheetah has to run.


A cheetah eats a variety of small animals, including game birds, rabbits, small antelopes (including the springbok, impala, and gazelle), young warthogs, and larger antelopes (such as the kudu, hartebeest, oryx, and roan). Their faces are distinguished by prominent black lines that curve from the inner corner of each eye to the outer corners of the mouth, like a well-worn trail of inky tears.


4 kg) Cheetah moms spend a lot of time teaching their cubs to chase, sometimes dragging live animals back to the den so the cubs can practice the chase-and-catch process


Advertisement If confronted, a roughly 125-pound cheetah will always run rather than fight -- it's too weak, light and thin to have any chance against something like a lion, which can be twice as long as a cheetah and weigh more than 400 pounds (181


Cheetahs eat a variety of small animals, including game birds, rabbits, small antelopes (including the springbok, impala, and gazelle), young warthogs, and larger antelopes (such as the kudu, hartebeest, oryx, and roan)


Historically, cheetahs ranged widely throughout Africa and Asia, from the Cape of Good Hope to the Mediterranean, throughout the Arabian Peninsula and the Middle East, from Israel, India and Pakistan north to the northern shores of the Caspian and Aral Seas, and west through Uzbekistan, Turkmenistan, Afghanistan, and Pakistan into central India. Header Links

LangSmith trace here

const chain4 = retrieveMap
  .assign({ context: formatDocs })
  .assign({ answer: answerChain })
  .pick(["answer", "docs"]);
// Note the documents have an article "summary" in the metadata that is now much longer than the
// actual document page content. This summary isn't actually passed to the model.
await chain4.invoke("How fast are cheetahs?");
{
  answer: [
    {
      answer: "\n" +
        "Cheetahs are the fastest land animals. They can reach top speeds of around 75 mph (120 km/h) and ro"... 74 more characters
    },
    { citations: [ { citation: [Array] }, { citation: [Array] } ] }
  ],
  docs: [
    Document {
      pageContent: "The maximum speed cheetahs have been measured at is 114 km (71 miles) per hour, and they routinely r"... 1048 more characters,
      metadata: {
        title: "cheetah - Encyclopedia Britannica | Britannica",
        source: "https://www.britannica.com/animal/cheetah-mammal",
        score: 0.97059,
        images: null
      }
    },
    Document {
      pageContent: "Contact Us − +\n" +
        "Address\n" +
        "Smithsonian's National Zoo & Conservation Biology Institute  3001 Connecticut"... 1343 more characters,
      metadata: {
        title: "Cheetah",
        source: "https://nationalzoo.si.edu/animals/cheetah",
        score: 0.95102,
        images: null
      }
    },
    Document {
      pageContent: "The science of cheetah speed\n" +
        "The cheetah (Acinonyx jubatus) is the fastest land animal on Earth, cap"... 738 more characters,
      metadata: {
        title: "How Fast Can a Cheetah Run?",
        source: "https://www.thoughtco.com/how-fast-can-a-cheetah-run-4587031",
        score: 0.94974,
        images: null
      }
    },
    Document {
      pageContent: "Now, their only hope lies in the hands of human conservationists, working tirelessly to save the che"... 880 more characters,
      metadata: {
        title: "How Fast Are Cheetahs, and Other Fascinating Facts About the World's ...",
        source: "https://www.discovermagazine.com/planet-earth/how-fast-are-cheetahs-and-other-fascinating-facts-abou"... 21 more characters,
        score: 0.92695,
        images: null
      }
    },
    Document {
      pageContent: "One of two videos from National Geographic's award-winning multimedia coverage of cheetahs in the ma"... 60 more characters,
      metadata: {
        title: "The Science of a Cheetah's Speed | National Geographic",
        source: "https://www.youtube.com/watch?v=icFMTB0Pi0g",
        score: 0.90754,
        images: null
      }
    },
    Document {
      pageContent: "The speeds attained by the cheetah may be only slightly greater than those achieved by the pronghorn"... 2527 more characters,
      metadata: {
        title: "Cheetah - Wikipedia",
        source: "https://en.wikipedia.org/wiki/Cheetah",
        score: 0.89476,
        images: null
      }
    }
  ]
}

LangSmith trace here

Generation post-processing​

Another approach is to post-process our model generation. In this example we’ll first generate just an answer, and then we’ll ask the model to annotate it’s own answer with citations. The downside of this approach is of course that it is slower and more expensive, because two model calls need to be made.

Let’s apply this to our initial chain.

import { StructuredTool } from "@langchain/core/tools";
import { formatToOpenAITool } from "@langchain/openai";
import { z } from "zod";

class AnnotatedAnswer extends StructuredTool {
  name = "annotated_answer";

  description =
    "Annotate the answer to the user question with quote citations that justify the answer";

  schema = z.object({
    citations: z
      .array(citationSchema)
      .describe("Citations from the given sources that justify the answer."),
  });

  _call(input: z.infer<(typeof this)["schema"]>): Promise<string> {
    return Promise.resolve(JSON.stringify(input, null, 2));
  }
}

const annotatedAnswerTool = formatToOpenAITool(new AnnotatedAnswer());

const llmWithTools5 = llm.bind({
  tools: [annotatedAnswerTool],
  tool_choice: annotatedAnswerTool,
});
import {
  ChatPromptTemplate,
  MessagesPlaceholder,
} from "@langchain/core/prompts";
import { RunnableSequence } from "@langchain/core/runnables";
import { JsonOutputKeyToolsParser } from "langchain/output_parsers";
import { RunnableMap, RunnablePassthrough } from "@langchain/core/runnables";
import { AIMessage, ToolMessage } from "@langchain/core/messages";

const prompt5 = ChatPromptTemplate.fromMessages([
  [
    "system",
    "You're a helpful AI assistant. Given a user question and some web article snippets, answer the user question. If none of the articles answer the question, just say you don't know.\n\nHere are the web articles:{context}",
  ],
  ["human", "{question}"],
  new MessagesPlaceholder({
    variableName: "chat_history",
    optional: true,
  }),
  new MessagesPlaceholder({
    variableName: "toolMessage",
    optional: true,
  }),
]);

const answerChain5 = prompt5.pipe(llmWithTools5);
const annotationChain = RunnableSequence.from([
  prompt5,
  llmWithTools5,
  new JsonOutputKeyToolsParser({
    keyName: "annotated_answer",
    returnSingle: true,
  }),
  (input: any) => input.citations,
]);
const map5 = RunnableMap.from({
  question: new RunnablePassthrough(),
  docs: retriever,
});
const chain5 = map5
  .assign({ context: formatDocs })
  .assign({ aiMessage: answerChain5 })
  .assign({
    chat_history: (input) => input.aiMessage,
    toolMessage: (input) =>
      new ToolMessage({
        tool_call_id: input.aiMessage.additional_kwargs.tool_calls[0].id,
        content: input.aiMessage.additional_kwargs.content ?? "",
      }),
  })
  .assign({
    annotations: annotationChain,
  })
  .pick(["answer", "docs", "annotations"]);
await chain5.invoke("How fast are cheetahs?");
{
  docs: [
    Document {
      pageContent: "They are surprisingly graceful\n" +
        "Cheetahs are very lithe-they move quickly and full-grown adults weigh"... 824 more characters,
      metadata: {
        title: "How Fast Are Cheetahs - Proud Animal",
        source: "https://www.proudanimal.com/2024/01/27/fast-cheetahs/",
        score: 0.96021,
        images: null
      }
    },
    Document {
      pageContent: "Contact Us − +\n" +
        "Address\n" +
        "Smithsonian's National Zoo & Conservation Biology Institute  3001 Connecticut"... 1343 more characters,
      metadata: {
        title: "Cheetah | Smithsonian's National Zoo and Conservation Biology Institute",
        source: "https://nationalzoo.si.edu/animals/cheetah",
        score: 0.94798,
        images: null
      }
    },
    Document {
      pageContent: "The science of cheetah speed\n" +
        "The cheetah (Acinonyx jubatus) is the fastest land animal on Earth, cap"... 738 more characters,
      metadata: {
        title: "How Fast Can a Cheetah Run? - ThoughtCo",
        source: "https://www.thoughtco.com/how-fast-can-a-cheetah-run-4587031",
        score: 0.92591,
        images: null
      }
    },
    Document {
      pageContent: "The maximum speed cheetahs have been measured at is 114 km (71 miles) per hour, and they routinely r"... 1048 more characters,
      metadata: {
        title: "Cheetah | Description, Speed, Habitat, Diet, Cubs, & Facts",
        source: "https://www.britannica.com/animal/cheetah-mammal",
        score: 0.90128,
        images: null
      }
    },
    Document {
      pageContent: "The Science of Speed\n" +
        "Instead, previous research has shown that the fastest animals are not the large"... 743 more characters,
      metadata: {
        title: "Now Scientists Can Accurately Guess The Speed Of Any Animal",
        source: "https://www.nationalgeographic.com/animals/article/Animal-speed-size-cheetahs",
        score: 0.90097,
        images: null
      }
    },
    Document {
      pageContent: "Now, their only hope lies in the hands of human conservationists, working tirelessly to save the che"... 880 more characters,
      metadata: {
        title: "How Fast Are Cheetahs, and Other Fascinating Facts About the World's ...",
        source: "https://www.discovermagazine.com/planet-earth/how-fast-are-cheetahs-and-other-fascinating-facts-abou"... 21 more characters,
        score: 0.89788,
        images: null
      }
    }
  ],
  annotations: [
    {
      sourceId: 0,
      quote: "We’ve mentioned that these guys can reach speeds of up to 70 mph, but did you know they can go from "... 22 more characters
    },
    {
      sourceId: 1,
      quote: "Built for speed, the cheetah can accelerate from zero to 45 in just 2.5 seconds and reach top speeds"... 52 more characters
    },
    {
      sourceId: 2,
      quote: "The maximum speed cheetahs have been measured at is 114 km (71 miles) per hour, and they routinely r"... 72 more characters
    }
  ]
}

LangSmith trace here

Help us out by providing feedback on this documentation page: