sammy

refresher

yesterday was the birth of Coco. to get a quick refresher or if you’re just seeing this project for the first time, catch up on the background knowledge in the original post.

technical details

this post is more about laying the technical foundation, and how we’ll operate moving forward. as a reminder, this was the rough high level architecture that is being used.

architecture

we started by revisiting Anthropic’s Effective harnesses for long-running agents Engineering blog post. at first glance, it seemed directly relevant since we were also building a very long running agent. the blog was geared more towards coding agents like claude code, where the key challenge is finding effective ways for agents to quickly understand the state of work when starting with a fresh context window.

the difference in our application is that our agent is only initialized once, and it basically runs throughout the lifetime of the product, producing an infinite stream of tokens. the blog addresses the cold start problem, which makes sense for coding agents that need to quickly orient themselves within a complicated codebase across many different tasks.

Coco works by just keeping one long context, and then continuously compacting when we hit a certain context length. compaction runs between messages sent when we hit a certain threshold (currently set to 60% of the context window). the last 10 items are kept verbatim, while the remainder of the context window is summarized using a separate model call.

beyond reacting to every slack message sent in the channel, Coco runs a proactive loop that checks every 5 minutes when idle. this way, it can follow up on pending work, check for new emails, or continue with ongoing tasks without needing to be continuously prompted.

tools

we have started with a very simple arsenal of tools for Coco to use:

workspace

run_shell: executing shell commands in workspace
read_file: read files from workspace
write_file: create/write files in workspace

slack

send_slack_message: post to channel

email

search_emails: gmail search with query syntax
read_email: fetch email content and download attachments
send_email: send with cc/reply/forward/attachments support

web

web_search: using Tavily search
fetch: extract content from a URL

while minimal, it is the initial toolset that Coco requires to be autonomous.

coco’s daily highlights

today involved a lot of testing and messing around with Coco to see what it was capable of.

tungsten cubes

inspired by Anthropic’s Project Vend, we tried to see if we could get any vendors to sell us tungsten cubes. we had Coco source vendors, and automatically draft an email to reach out and get a quote. to our pleasant surprise, a gentleman from China responded promptly. he was even nice enough to include an image of the tungsten he had available:

we instructed Coco to act with some urgency, so it responded with this email:

the interaction went as far as getting the quote and shipping details, before we abandoned the tungsten route as our validation of Coco being able to act autonomously.

vision

Coco needs to read tech packs to communicate with vendors. our ingestion only grabbed text, so Coco couldn’t see images embedded in the documents.

we gave it a read_image tool that lets Coco view images in its workspace. when an image file is downloaded (from email attachments, for example), Coco can read it and visually process the contents.

coco recognizes coconuts

Coco also showed us its quirky side today:

next steps involve getting ready for scale. it’s time to begin the process of reaching out to manufacturers based on the swimsuit specifications that Coco now understands.

we are also planning an architectural change to leverage subagents, along with new function tools such as browser access (to fill out forms) and audio capabilities (to make phone calls).

coco token tracker

and to finish off, here is the daily token tracker. Coco was once again hard at work consuming ~20 million tokens, costing us about $102:

be sure to follow me on X for more updates!

project coco 02: tungsten cubes