When an agent determines it needs to invoke a function (commonly called "tools" nowadays), the conventional approach is to invoke a function within the same process as the agent.

This is a good starting point for simple tools, written in the same language as the agent orchestration, but as systems get larger they tend to involve multiple services or other constraints that would make this infeasible.

An initial approach may be making an API call to the other service. However, this method creates a few challenges:

1. Internal tools become exposed to the public internet
2. Operations work is required to enable ingress to the tool
3. Authentication mechanisms must be implemented to prevent malicious access
4. Network ingress needs to handle load balancing between multiple tool instances
5. HTTP timeouts mean that the agent can't execute a tool that takes longer than the HTTP timeout
6. Avoiding the HTTP timeout means that you have to implement your own internal job queue
7. HTTP server needs to be scaled to handle the peak concurrency of the tool

...and I could go on. So, no - in our humble opinion - we don't think that's a good enough way to build production grade agentic workflows.

## What if we could run our tools without exposing them to the public internet?

This reality - and our own experience of building distributed systems - drove our early architectural decision to build Inferable in a way that requires:

1. Zero network ingress to a VPC
2. No additional custom authentication to make an Inferable service operational

**Enter (long) polling.**

Essentially, when a function gets registered (either in the codebase directly or via a proxy service), rather than keeping a port open, the SDK periodically sends a heartbeat to the control plane saying - "Hey, I'm alive, and I'm capable of running these functions. Do you have anything for me"?

The control plane then responds with either:

1. Nothing for now, keep ticking.
2. Yes, execute these functions for me, and these are the input params. When you're done, call me back on this address.

## Distributed System Guarantees

The inferable execution engine implements a distributed job queue (something like SQS, but especially designed for these workloads) which makes sure:

1. **Message Visibility**: Jobs cannot be processed by multiple machines simultaneously through our locking mechanism
2. **At-least-once Processing**: A job will be processed at least once as long as a capable machine is available
3. **Timeout Management**: If a machine exceeds timeout or stalls, the job is automatically rerouted to a different replica
4. **Exclusive Processing**: Only one machine can process a job at any time via a time-limited lease
5. **Result Authentication**: Only the machine with the acquired lease can submit results
6. **Load Distribution**: Workload is balanced across multiple replicas running the same service configuration

## Implementation: PostgreSQL-Based Job Queue

Rather than using a traditional message queue, we've implemented our distributed job queue using PostgreSQL. We did this because - well, why not? [All you need is postgres](https://www.amazingcto.com/postgres-for-everything/), and you get:

1. Transactional guarantees
2. Familiar operational characteristics for most engineering teams for self-hosting

Here's a look at the core polling logic:

```sql
UPDATE jobs SET
  status = $1,
  remaining_attempts = remaining_attempts - 1,
  last_retrieved_at = now(),
  executing_machine_id=$2
WHERE id IN (
  SELECT id FROM jobs
  WHERE
    status = 'pending'
    AND cluster_id = $3
    AND service = $4
  LIMIT $5 -- limit of jobs to claim
  FOR UPDATE SKIP LOCKED
```

And here's how a job moves between states:

```

Machine claims job

Job completes

Job fails

Timeout/Machine failure

Retry available

No retries left

pending

running

success

failure

stalled
```

Our job queue implementation handles all of the above scenarios, and a few more that we need to get right in order to make this work at scale.

### 1\. Job Claiming

We use PostgreSQL's `SELECT FOR UPDATE SKIP LOCKED` pattern to ensure atomic job claiming:

```typescript
// Inside pollJobs function
SELECT id FROM jobs
WHERE
  status = 'pending'
  AND cluster_id = ${clusterId}
  AND service = ${service}
LIMIT ${limit}
FOR UPDATE SKIP LOCKED
```

### 2\. Approval Workflows

Some functions require explicit approval before execution. And a human approval is done out of band and we should be able to pause the job in-queue to wait for approval. See [human in the loop](https://docs.inferable.ai/pages/human-in-the-loop) for more details.

This is not something we can implement easily via synchronous HTTP calls, as approvals may exceed the HTTP timeout.

```typescript
export async function submitApproval({
  call,
  clusterId,
  approved,
}: {
  call: NonNullable<Awaited<ReturnType<typeof getJob>>>;
  clusterId: string;
  approved: boolean;
}) {
  if (approved) {
    await data.db
      .update(data.jobs)
      .set({
        approved: true,
        status: "pending",
        executing_machine_id: null,
        remaining_attempts: sql`remaining_attempts + 1`,
      })
      .where(/* conditions */);
  } else {
    // Handle rejection...
  }
}
```

### 3\. Job Recovery

The system automatically recovers from machine failures:

```sql
-- run periodically
UPDATE jobs SET
  status = 'pending',
  executing_machine_id = null
WHERE
  status = 'stalled'
  AND cluster_id = $1
  AND service = $2
```

## How does this work in practice?

Consider running a Kubernetes cluster with a `user-service` comprising 20 pods, each registering a `getUser()` function. If we receive 100 `getUser()` requests, our system ensures:

1. Under normal conditions, exactly 100 executions occur across the 20 pods
2. In failure scenarios, all requests are processed at least once (at least 100 executions)
3. Requests are distributed to avoid overwhelming any single pod
4. If a pod fails, pending requests are automatically reassigned

The beauty of this approach is that it maintains security through isolation while ensuring reliable function execution, all without requiring developers to implement complex distributed systems patterns or security measures themselves.

By using PostgreSQL as our job queue, we get the benefits of a battle-tested database while maintaining the flexibility to implement complex workflows. The `SELECT FOR UPDATE SKIP LOCKED` pattern ensures that our distributed system maintains consistency even under heavy load with multiple consumers.

All of this is what a developer gets for free when they register a function like this:

```javascript
service.register({
  func: getUser,
});
```

If you find this interesting, all of our source code is [open source and available on GitHub](https://github.com/inferablehq/inferable). Also, as long as you can provision a postgres instance, you can [self-host Inferable](https://docs.inferable.ai/pages/self-hosting).