Skip to main content

ChatVertexAI

LangChain.js supports Google Vertex AI chat models as an integration. It supports two different methods of authentication based on whether you're running in a Node environment or a web environment.

Setup​

Node​

To call Vertex AI models in Node, you'll need to install the @langchain/google-vertexai package:

tip

See this section for general instructions on installing integration packages.

npm install @langchain/google-vertexai
tip

We're unifying model params across all packages. We now suggest using model instead of modelName, and apiKey for API keys.

You should make sure the Vertex AI API is enabled for the relevant project and that you've authenticated to Google Cloud using one of these methods:

  • You are logged into an account (using gcloud auth application-default login) permitted to that project.
  • You are running on a machine using a service account that is permitted to the project.
  • You have downloaded the credentials for a service account that is permitted to the project and set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of this file.
tip

See this section for general instructions on installing integration packages.

npm install @langchain/google-vertexai

Web​

To call Vertex AI models in web environments (like Edge functions), you'll need to install the @langchain/google-vertexai-web package:

npm install @langchain/google-vertexai-web

Then, you'll need to add your service account credentials directly as a GOOGLE_VERTEX_AI_WEB_CREDENTIALS environment variable:

GOOGLE_VERTEX_AI_WEB_CREDENTIALS={"type":"service_account","project_id":"YOUR_PROJECT-12345",...}

Lastly, you may also pass your credentials directly in code like this:

import { ChatVertexAI } from "@langchain/google-vertexai-web";

const model = new ChatVertexAI({
  authOptions: {
    credentials: {"type":"service_account","project_id":"YOUR_PROJECT-12345",...},
  },
});

Usage​

The entire family of gemini models are available by specifying the modelName parameter.

For example:

import { ChatVertexAI } from "@langchain/google-vertexai";
// Or, if using the web entrypoint:
// import { ChatVertexAI } from "@langchain/google-vertexai-web";

const model = new ChatVertexAI({
  temperature: 0.7,
  model: "gemini-1.0-pro",
});

const response = await model.invoke("Why is the ocean blue?");
console.log(response);
/*
AIMessageChunk {
  content: [{ type: 'text', text: 'The ocean appears blue due to a phenomenon called Rayleigh scattering. This occurs when sunlight' }],
  additional_kwargs: {},
  response_metadata: {}
}
 */

API Reference:

tip

See the LangSmith trace for the example above here.

Streaming​

ChatVertexAI also supports streaming in multiple chunks for faster responses:

import { ChatVertexAI } from "@langchain/google-vertexai";
// Or, if using the web entrypoint:
// import { ChatVertexAI } from "@langchain/google-vertexai-web";

const model = new ChatVertexAI({
  temperature: 0.7,
});
const stream = await model.stream([
  ["system", "You are a funny assistant that answers in pirate language."],
  ["human", "What is your favorite food?"],
]);

for await (const chunk of stream) {
  console.log(chunk);
}

/*
AIMessageChunk {
  content: [{ type: 'text', text: 'Ahoy there, matey! Me favorite grub be fish and chips, with' }],
  additional_kwargs: {},
  response_metadata: { data: { candidates: [Array], promptFeedback: [Object] } }
}
AIMessageChunk {
  content: [{ type: 'text', text: " a hearty pint o' grog to wash it down. What be yer fancy, landlubber?" }],
  additional_kwargs: {},
  response_metadata: { data: { candidates: [Array] } }
}
AIMessageChunk {
  content: '',
  additional_kwargs: {},
  response_metadata: { finishReason: 'stop' }
}
*/

API Reference:

tip

See the LangSmith trace for the example above here.

Tool calling​

ChatVertexAI also supports calling the model with a tool:

import { ChatVertexAI } from "@langchain/google-vertexai";
import { type GeminiTool } from "@langchain/google-vertexai/types";
import { zodToGeminiParameters } from "@langchain/google-vertexai/utils";
import { z } from "zod";
// Or, if using the web entrypoint:
// import { ChatVertexAI } from "@langchain/google-vertexai-web";

const calculatorSchema = z.object({
  operation: z
    .enum(["add", "subtract", "multiply", "divide"])
    .describe("The type of operation to execute"),
  number1: z.number().describe("The first number to operate on."),
  number2: z.number().describe("The second number to operate on."),
});

const geminiCalculatorTool: GeminiTool = {
  functionDeclarations: [
    {
      name: "calculator",
      description: "A simple calculator tool",
      parameters: zodToGeminiParameters(calculatorSchema),
    },
  ],
};

const model = new ChatVertexAI({
  temperature: 0.7,
  model: "gemini-1.0-pro",
}).bind({
  tools: [geminiCalculatorTool],
});

const response = await model.invoke("What is 1628253239 times 81623836?");
console.log(JSON.stringify(response.additional_kwargs, null, 2));
/*
{
  "tool_calls": [
    {
      "id": "calculator",
      "type": "function",
      "function": {
        "name": "calculator",
        "arguments": "{\"number2\":81623836,\"number1\":1628253239,\"operation\":\"multiply\"}"
      }
    }
  ],
}
 */

API Reference:

tip

See the LangSmith trace for the example above here.

withStructuredOutput​

Alternatively, you can also use the withStructuredOutput method:

import { ChatVertexAI } from "@langchain/google-vertexai";
import { z } from "zod";
// Or, if using the web entrypoint:
// import { ChatVertexAI } from "@langchain/google-vertexai-web";

const calculatorSchema = z.object({
  operation: z
    .enum(["add", "subtract", "multiply", "divide"])
    .describe("The type of operation to execute"),
  number1: z.number().describe("The first number to operate on."),
  number2: z.number().describe("The second number to operate on."),
});

const model = new ChatVertexAI({
  temperature: 0.7,
  model: "gemini-1.0-pro",
}).withStructuredOutput(calculatorSchema);

const response = await model.invoke("What is 1628253239 times 81623836?");
console.log(response);
/*
{ operation: 'multiply', number1: 1628253239, number2: 81623836 }
 */

API Reference:

tip

See the LangSmith trace for the example above here.

VertexAI tools agent​

The Gemini family of models not only support tool calling, but can also be used in the Tool Calling agent. Here's an example:

import { z } from "zod";

import { DynamicStructuredTool } from "@langchain/core/tools";
import { AgentExecutor, createToolCallingAgent } from "langchain/agents";

import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatVertexAI } from "@langchain/google-vertexai";
// Uncomment this if you're running inside a web/edge environment.
// import { ChatVertexAI } from "@langchain/google-vertexai-web";

const llm: any = new ChatVertexAI({
  temperature: 0,
});

// Prompt template must have "input" and "agent_scratchpad input variables"
const prompt = ChatPromptTemplate.fromMessages([
  ["system", "You are a helpful assistant"],
  ["placeholder", "{chat_history}"],
  ["human", "{input}"],
  ["placeholder", "{agent_scratchpad}"],
]);

const currentWeatherTool = new DynamicStructuredTool({
  name: "get_current_weather",
  description: "Get the current weather in a given location",
  schema: z.object({
    location: z.string().describe("The city and state, e.g. San Francisco, CA"),
  }),
  func: async () => Promise.resolve("28 °C"),
});

const agent = await createToolCallingAgent({
  llm,
  tools: [currentWeatherTool],
  prompt,
});

const agentExecutor = new AgentExecutor({
  agent,
  tools: [currentWeatherTool],
});

const input = "What's the weather like in Paris?";
const { output } = await agentExecutor.invoke({ input });

console.log(output);

/* 
It's 28 degrees Celsius in Paris.
*/

API Reference:

tip

See the LangSmith trace for the agent example above here.

Help us out by providing feedback on this documentation page: