We recently released the [v2 of the Teams AI Library](https://microsoft.github.io/teams-ai/welcome)—a new take on building bots for the Microsoft Teams ecosystem, with a focus on better developer experience.

Our team is small—just 5 engineers—tasked with building out the TypeScript SDK, .NET SDK (because Microsoft), and Python SDK. On top of that, we’re also responsible for writing and maintaining the documentation for all three.

We initially tried using AI to generate documentation by feeding it the code and some sample projects. While it got us started, the quality wasn’t good enough to publish. So we ended up writing all our docs by hand. That’s right—hand-crafted, artisanal docs. Who would've thought, in 2025.

The library got a fair amount of [attention](https://devblogs.microsoft.com/microsoft365dev/announcing-the-updated-teams-ai-library-and-mcp-support) during Microsoft Build—with a shoutout from Satya in the opening keynote and a live demo in the second. That’s one of the wild things about working at Microsoft: if your project gets the right visibility, interest and usage can spike overnight.

But that visibility comes at a cost—**support**.

We’re now spending a good chunk of time responding to questions. These tend to fall into a few buckets:

1. How do I do something? - This might be covered in the docs—or not. 
2. There’s a mistake or gap in the docs.
3. It’s a scenario we could support, but haven’t tested yet.
4. It’s a totally new use case that we don’t currently support, but maybe we should.

This support burden has made me think about how we can build a better system using agents to close the feedback loop between our users, our documentation, and our roadmap.

Here’s how I imagine agents helping:

1. Surfacing existing answers: If the answer already exists in the docs, the agent can quickly surface it. 
 _Example: “How can I use adaptive cards in TypeScript?”_

2. Filling in documentation gaps: If the answer isn’t documented, but we help the user figure it out in chat or email, we should be able to trigger the agent to propose an update to the docs—either in an FAQ or directly in the core pages.

3. Logging new scenarios: If it’s a brand new use case, we should be able to ask the agent to file a GitHub issue describing it. Once there’s enough interest, we can prioritize it.

My goal is to build this system out—as both a real tool for our team and a case study in using single- or multi-agent architectures in the wild. 

## Basic architecture

I view this problem in 3 parts - 
```mermaid
flowchart TD
 A[User Conversation] --> B[Doc Agent]
 B --> C[Git Agent]
```
1. The user conversation is the place where the conversation between us and the users takes place. This happens to be Teams in my case, but this could easily be in other places too.
2. The doc agent - this agent is responsible for all the doc-related things. Its responsibilities could include querying docs that exist or making updates to it
3. The git agent - if the doc agent decides to make any updates to the docs, those updates will need to be committed back to git. Ideally, this should go through a pull-request process where a human could make adjustments or approve the updates.

This architecture allows us to expand the capabilities of each area to beyond what we need immediatly. For example, I can imagine the doc agent also being invoked directly from the PR to make adjustments (instead of me manually fixing some issue with the PR). Additionally, the git-agent could be used for a variety of other use-cases beyond just documentation because it works generally with Git.

## Interacting with the agent

I decided to use this opportunity to dogfood our [Typescript Library](https://microsoft.github.io/teams-ai/typescript/getting-started). My secondary goal is to eventually use our multi-agent plugins (I recently added an [A2A plugin](https://microsoft.github.io/teams-ai/typescript/in-depth-guides/ai/a2a/)) to see if there are any actual improvements to these new protocols. 

Also, I'm just extremely comfortable with the integration layer with Teams here, and that's where the support questions come in right now, so this keeps things simple. In the future, if the support questions start coming in to a different communication channel, I can reconsider this decision.

## Doc Agent

I took inspiration from [Thornston Ball](https://x.com/thorstenball/)'s [How to Build An Agent](https://ampcode.com/how-to-build-an-agent) for this agent. In his blog post, he describes in beautiful detail how he added simple tool-calls to his coding agent to give this agent the ability to understand and update code. 

Before reading this, I would have definitely leaned on building an RAG layer using embeddings or something more complicated. But I decided to keep things very simple and use straightforward tools to do this.

```mermaid
%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%%
flowchart TD
 Doc[Doc Agent] --> LLM
 subgraph llm_tools
 LLM --> list[list_files]
 LLM --> read[read_file]
 LLM --> edit[edit_file]
 end
 list --> |list all the files| docs_md
 docs_md --> |ls -r files and file-paths| list
 read --> |read fileX| docs_md
 docs_md --> |content from fileX| read
 edit --> |update fileX with content| docs_md
 docs_md --> |✅| edit
 subgraph docs_md[documentation]
 md[markdown docs]
 end
```
With just the above, I had a fairly reasonable agent that was able to answer some rudimentary questions from my documentation. 

![Doc agent answering questions](https://github.com/heyitsaamir/blog/blob/main/public/large-files/answering_doc_agent.gif?raw=true)

You can see the agent is able to execute the tools I gave it—looking up and answering questions effectively. Right now, it only uses the file names as hints to decide what to read, so an obvious optimization is to provide richer metadata alongside each file. That way, the agent can make more informed decisions about what to read and when.

One nice detail: at the beginning, it calls the list_files tool, but in later requests, it doesn’t need to because the result is cached in its context memory. The downside is that this memory is pretty naïve—it stores the entire content of each file it reads and keeps sending that back to the agent. A smarter approach might be for it to remember summaries like “I read this file and it was about XYZ,” and only re-read the file if it actually needs to.

Okay, so with this we have a basic doc agent working. It can read files, and help look up relevant documentation based on your query. 

### Editing files

Let's take a more closer look at the `edit_files` tool. The tool definition looks something like this:
```ts
 {
 name: "edit_file",
 description: "Edit the contents of a given relative file path. Use this when you want to change what's inside a file. If the path does not exist, it will be created.",
 parameters: {
 type: 'object',
 properties: {
 path: {
 type: 'string',
 description: 'The relative path of a file in the working directory.',
 },
 content: {
 type: 'string',
 description: 'The new content of the file.',
 },
 },
 required: ['path', 'content'],
 },
 execute: async function ({ path: incomingPath, content }: { path: string, content: string }) {
 console.log('Executing edit_file with path: ', incomingPath);
 const basePath = process.cwd();
 const baseDirectory = path.join(basePath, 'md');
 const fullPath = path.join(baseDirectory, incomingPath);
 // If the directory does not exist, create it
 const directory = path.dirname(fullPath);
 if (!fs.existsSync(directory)) {
 fs.mkdirSync(directory, { recursive: true });
 }
 // If the file does not exist, create it
 fs.writeFileSync(fullPath, content);
 return 'File updated';
 }
}
```
We essentially ask the LLM to give us the updated content of the file (or if it doesn't exist, the new content of the file). Then we simply replace the content of the file.

Now this is fine for a prototype, and it works, but there are a few problems with this:
1. We can't be updating the source documentation because it's not verified by a human. If the LLM has the ability to change up the source-documentation willy-nilly, then it'll use this un-verified documentation for subsequent lookups. 
2. This leads to further considerations on which _working copy_ of the documentation the agent should use. 

Let's keep these issues in the back of our minds as we proceed to the next part of the problem.

## Git Agent

Okay, the next part of the workflow is for us to bring these changes to git where these edits are created in an independent branch, pushed to remote, and and create a PR for a human to review. This has several important benefits:
1. Use git for storage - normally, for each branch, we'd need to save copies of changes somewhere. Using git, this all comes basically for free.
2. Human-In-Loop - we need some known natural mechanism for humans to come in and review the changes that are being suggested. What better than the review system that Github has built that we use to review our code.
3. Separation of changes - having separate branches allows us to have logical separation in changes in documentation that the agent is suggesting. This is just a natural part of Git itself.

To build this, I decided to build this as a separate agent altogether. The responsibility of this agent was simply to interact with Git. It did not have to be an expert on documentation at all, but simply to manage Git related activities. 

```mermaid
%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%%
flowchart TD
 GitAgent[Git Agent] --> LLM
 subgraph llm_tools
 LLM --> clone[clone_repo]
 LLM --> checkout[checkout_branch]
 LLM --> apply[apply_changes]
 LLM --> commit[commit_and_push]
 LLM --> pr[create_pr]
 end

 clone --> |clone repo| tempRepo[Temporary Repo]
 tempRepo --> |local path| clone

 checkout --> |checkout new branch| tempRepo
 tempRepo --> |branch ready| checkout

 apply --> |edit/create files| tempRepo
 tempRepo --> |changes applied| apply

 commit --> |commit & push| tempRepo
 tempRepo --> |✅ pushed| commit

 pr --> |create pull request| GitHub
 GitHub --> |PR URL| pr

 subgraph GitHub
 repo[Remote Repo]
 end
```

In practice, it uses: 
 - [Octokit](https://github.com/octokit) which is Github's API. This API is necessary to work with Github where the docs are all stored and where PRs would be created and managed. 
 - [SimpleGit](https://www.npmjs.com/package/simple-git) which is a light weight package that brings git-commands to node processes. This allows us to run commands like `git.clone` or `checkoutLocalBranch`

Testing this agent out independently from the Doc Agent results in this sort of interaction:

| <img width="1074" alt="Image" src="https://github.com/user-attachments/assets/55c4c61e-1d68-4542-a621-276c6108b11b" /> |
|:--:| 
| <img width="1002" alt="Image" src="https://github.com/user-attachments/assets/c269c9ba-f169-4e4e-b2a6-e4e0603b70d1" /> | 
| *The resulting pull-request on github* |

## Combining the two agents together

Now we have the two agents working independently. The next step is for us to combine the two agents together so they can work together. Here is the rough interaction I'm aiming for:

```mermaid
sequenceDiagram
 participant U as User
 participant DA as Doc Agent
 participant GA as Git Agent
 participant GH as Github

 U->>DA: What does the documentation say about X?
 DA->>GA: checkForBranch(threadId)
 GA->>GH: Check if branch exists for thread
 GH-->>GA: Branch exists? (Yes/No)
 alt Branch exists
 GA-->>DA: Use branch for threadId
 else No branch
 GA-->>DA: Use main branch
 end
 DA->>U: The documentation says Y about X.

 U->>DA: Update the documentation on X to include Z additional details
 DA->>U: Here are the changes I'm going to make. Does this make sense?
 U->>DA: Yes that makes sense
 DA->>GA: applyChanges(threadId, [{path, content}])
 GA->>GH: Push new branch with changes
 GH-->>GA: Branch pushed confirmation
 GA-->>DA: "Change applied" to a new branch pushed to remote
 DA->>U: The changes are committed to a new branch. Should I create a new PR?
 U->>DA: Yes go for it.

 U->>DA: "Create a PR"
 DA->>GA: createPR(threadId, title, body, base)
 GA->>GH: Open pull request with changes
 GH-->>GA: PR created with URL
 GA-->>DA: "PR created: <url>"
 DA-->>U: "PR created: <url>"
```

You'll notice that I'm using the `threadId` as the key for each documentation change here. This is because logically, I decided that each PR would be for a particular support thread. This allows us to attribute changes back to a particular thread, and usually a particular thread adheres to one support topic, so PRs should be constrained to one area. We can always revisit this decision in the future, but for now this works out.

Introducing a working copy concept requires coordination between the doc-agent and git-agent to read from or interact with a specific state of a repository. There are four possible strategies for managing this integration, each with trade-offs in responsibility and architectural complexity:
1.	**Shared Filesystem Approach** - If git-agent and doc-agent are co-located on the same machine, the doc-agent can directly access the working copy (e.g., at a path like /tmp/branch-X) and read from it. This approach is simple but tightly couples deployment and assumes shared file access.
2.	**Delegated Access via git-agent** - In this model, doc-agent requests data from git-agent through APIs like “list files” or “read file contents.” This centralizes git-related logic in one place but increases the responsibility of git-agent significantly, turning it into a file-serving proxy.
3.	**Independent Git Logic in doc-agent** - Here, the doc-agent manages its own working copy by performing git operations directly. While this preserves separation of concerns, it duplicates logic across agents and increases maintenance overhead.
4.	**Tool Sharing via MCP (Hybrid Approach)** - A balanced solution is to keep git operations inside git-agent, but expose reusable tools (e.g., readFileFromRepo) via a shared interface like MCP. Other agents, including doc-agent, can invoke these tools without git-agent having to orchestrate the request directly. This encourages reusability and keeps agent responsibilities focused.

```mermaid
%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%%
graph TD
 A[doc-agent] -->|Read files directly| B["Shared path (e.g., /tmp/branch-X)"]
 A -->|API requests| C[git-agent]
 C -->|Responds with file content, file list| A
 A -->|Calls shared tools| E[git-agent exposes tools via MCP]
 A -->|Performs own git operations| D[(Local git logic)]
 E -->|Re-uses git logic| F[(Internal git operations)]

 subgraph Strategy 4
 E --> F
 end
 subgraph Strategy 3
 D
 D-->|clone, checkout, read, list|I[tmp git folder]
 end
 subgraph Strategy 2
 C
 C --> |clone, checkout| H[tmp git folder]
 C --> |list, read| H
 end
 subgraph Strategy 1
 B
 G[git-agent] --> B
 end
```

For this current prototype stage, I decided to stick with option 1 since all the agents are together in one single deployment. I want to take this first phase and see how the agent does, but I suspect I'll be moving toward strategy 4 soon after.

## Next Steps

<iframe width="560" height="315" src="https://www.youtube.com/embed/mLg02sBcZ74?si=52is-6VvmW50arjY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

I’ve put together a working prototype of the doc-agent at https://github.com/heyitsaamir/docs-agent. This is very much a **work in progress**. The core ideas are in place, but many pieces are still evolving—including how it interacts with a working copy, how memory is scoped and retrieved, and how it behaves across different types of documentation tasks.

As I use the agent in more real-world workflows, I’ll be refining the architecture and sharing deeper dives on specific areas like tool composition, memory shaping, and multi-agent collaboration. Stay tuned!

Improving the Documentation Feedback Loop

Muslims are expected to pray 5 times a day. The times for the start of each prayer depend on the angle of the sun relative to where you life. So these are constantly changing depending on the the location and time of the day1.

Before smart phones, people would have calendars on their fridges with prayer times. You'd have a general idea of what those timings were and you'd regularly scan these if you remembered to update your estimates. 

Once smart phones became ubiquitous, everyone had apps that would help them keep tabs on what these timings were. There are literally hundreds of apps out there that do this2. Personally, I use an app called [Pillars](https://www.thepillarsapp.com/) which, unlike the other apps, is completely ad-free, and is focused on absolutely no data-tracking whatsoever.

But, I spend most of my day in front of my laptop. And I've worked fairly hard to break my dependency on my smartphone (see [Addiction to my phone](https://www.aamirj.com/posts/9_addiction-to-my-phone)). I needed a solution for me to help me keep track of these times on my mac, so I built a simple little solution for myself.

[SalatTimesBar](https://heyitsaamir.github.io/SalatTimesBar/)

![Image](https://github.com/user-attachments/assets/58745003-58c2-45d6-9b7e-3ab3bf65d039)

It took me about a week to build on-and-off. But over 2 years to release it. 

When I was on my paternity leave, I got some time to build this out as a way to not only solve this problem, but also learn Swift UI. I was inspired by the simplicity of the [ItsyCal](https://www.mowglii.com/itsycal/) menubar app, which I use every day to track my calendar. It's such a simple elegant solution and I wanted to build just that.

The features are simple:
- for a given location, fetch the salat times (I used [AlAthan](https://aladhan.com/prayer-times-api), but honestly, I could have just built a calculation library based on 1)
- display it in a simple way

Given that, the actual app logic was really easy to build. The hard parts were:
- Publishing: Apple does not make this easy. I can build apps for my computer, but I can't distribute them. I really wanted to build this solution for my personal macbook and my work one. And as a bonus, I wanted to give away this solution to all. The outcome of this was to register a developer account ($99/year, really, Apple?), and submit it for notarization before distribution so that all macs know that it's been verified by Apple.
- Figuring out how to get automatic updates working - Once distributed, I needed to make sure that further bug-fixes or features could be automatically installed for all users. I wanted to distribute this app via Github (honestly, I think I might be okay just going via the app store now), and if there were any severe errors, I wanted users to not have to re-download updates. There's a popular open source solution to do this (https://sparkle-project.org/) but it wasn't trivial to set up. Essentially, it relies on an RSS feed to compare semver versions, and download and replace the existing installation on a user's machine. The library handles lots of logic like how to check, how often to check, how to update etc. 

But now that it's all set up, I announced it in a few places and it's been really fun seeing the appreciation and love for it :). 

Also, now that I have an apple developer account for the year, I should think about building some iOS or other mac apps :)


---
1 - More details on how these are calculated [here](https://praytimes.org/calculation).
2 - Muslim Pro, Athan, Pillars are some of the popular apps out there.

Keeping track of Salat Times - on your mac!

For a hackathon at work, I finally decided to build something with AI agents. For the last hackathon, I had started to experiment with embeddings in the previous Hasan where I had built a simple wiki search tool. But this time I really wanted to start experimenting with LLM agents in a practical application.

A team member had done a presentation at work describing the recent rapid advancements in AI image reasoning. The talk was very interesting and I found myself going through the various sources and articles he had linked. I forked [this](https://github.com/ishan0102/vimGPT) repo and started to experiment.

# Basics

The basic structure for the existing project was as follows:

```mermaid
sequenceDiagram
 User->>+Orchestrator:Sends objective
 Orchestrator->>+Browser Layer:Initializes Browser
 loop Loops until objective is achieved
 Browser Layer->>-Orchestrator:Sends browser screenshot
 Orchestrator->>+AI Layer:Sends Objective with screenshot
 AI Layer->>+Vision Layer:Builds prompt
 Vision Layer->>+OpenAI:Sends prompt and screenshot
 OpenAI->>-Vision Layer:Returns action to be performed
 Vision Layer->>-Orchestrator:Returns action to be performed
 Orchestrator->>-Browser Layer:Execute action
 end
```

Essentially, the Vision layer would send a screenshot to the AI agent and ask it to return the instructions for the next action. Actions included things like clicking, typing or navigating. Alternatively, if the AI determined that the objective was complete, it could return a signal that would terminate the program. The browser layer was a wrapper on top of [Playwright](https://playwright.dev/) which spawns a new browser and can be controlled by the program.

## Vimium chrome extension

The forked project made use of [Vimium Chrome Extension](https://chromewebstore.google.com/detail/vimium/dbepggeogbaibhgnhhndojpepiihcmeb?pli=1) as a clever way to highlight interactable elements on the screen. The prompt asks the agent to return the id of the elements that it should use in the next action. This keeps the agent focused on what actions it _can_ actually perform.
<img width="364" alt="image" src="https://github.com/heyitsaamir/blog-next-js/assets/48929123/635de7a6-040d-4203-aed4-7639e361d5d7">

Ofcourse, this is not fool-proof. Websites often do weird things with their sites where certain divs that are actually interactable don't have accessible roles or are interactable through javascript. This reduces the effectiveness of the chrome extension in those situations. More on that later.

I also modified the extension myself for my use case so it would _see_ some more types of elements and would not rely on a keyboard shortcut to trigger hint-mode. My fork is [here](https://github.com/heyitsaamir/vimium) if you're interested.

# Observations and Ideas of improvement

The basic operation of this project was surprisingly simple. The AI model was surprisingly good at making simple decisions given.an objective and a screenshot of a webpage. However, for many different reasons, it struggled, and it struggled a lot. Below, I've documented a number of these instances and some strategies I used to get past them (or not).

## Sticking to a single app

I decided to stick with a single app for my experiments. This allowed me to have better control over the environment and not have too much variability. Its true that our tool should be able to handle unpredictability with ease, but I wanted it to graduate to those use cases eventually. Ideally, I should have built my own "realistic" website which would let me have even more control over the experiments, but I decided to skip this step for now and go with something that was already built.

The website I chose was [Todoist](https://todoist.com/). The main reasons were:
1. Generally when you're experimenting with any new tech, after a basic Hello World app, you try to build a simple to do app.
2. It was a realistic use case which could inspire automation. Users could ask the agenet to go out and add a todo item, or retrieve them.
3. Its interface was clean and accessible. Many blogs and wordpress sites are often littered with ads and popups (Which can introduce more unpredictability). The interface for Todoist is simple and without much noise. 
<img width="751" alt="image" src="https://github.com/heyitsaamir/blog-next-js/assets/48929123/2c2ed046-03ec-40bf-be42-5fe212e72986">

Some sample objectives I played around with were:
1. Add a task called Do Laundry to my task list
2. Retrieve all my tasks from the task list


![ezgif-6-b8b8997eee](https://github.com/heyitsaamir/blog-next-js/assets/48929123/1abfdc24-3df7-4b65-8fea-7540b737904d)
An example of what a typical session looked like. The program takes a screenshot and sends it to the agent which decides on the next action until the objective is complete.

## Unending sessions

Initially, the prompt I had forked looked something along the lines of this:
> You need to choose which action to take to help a user do this task: {objective}. Your options are navigate, type, click, and done. Navigate should take you to the specified URL. Type and click take strings where if you want to click on an object, return the string with the yellow character sequence you want to click on, and to type just a string with the message you want to type. For clicks, please only respond with the 1-2 letter sequence in the yellow box, and if there are multiple valid options choose the one you think a user would select. For typing, please return a click to click on the box along with a type with the message to write. **When the page seems satisfactory, return done as a key with no value.** You must respond in JSON only with no other fluff or bad things will happen. The JSON keys must ONLY be one of navigate, type, or click. Do not return the JSON inside a code block.

Notice the "When the page seems satisfactory, return done as a key with no value". The problem with keeping the prompt fairly open and the completion condition be along the lines of "when you think the objective is complete" is that sometimes it's not clear to the model that the objective was successfully completed. In my experiments, sometimes the model would continue to try to add tasks even though it was successfully able to do it. It wasn't clear what the meaning of "satisfactory" really is.

To get past this, I introduced a `completion_condition` parameter (alongside the `objective` parameter). This seemed to work fairly well. With an explicit completion condition, the model seemed to have a clear understanding of what "done" meant and the un-ending session problem seemed to have been solved.

## Authentication

Ah authentication. I decided to simply punt on this problem. Yes authentication will be something that needs to be solved, but it's boring (but important), so I decided to tackle it later. Atleast until I was more confident in the core tech.
For this reason, I decided to sign in to the chromium browser and use a persistent browser context as opposed to a non-persistent one.

## Desktop vs Mobile

In the same presentation mentioned above, one of the slides referenced [this](https://arxiv.org/abs/2401.16158) paper titled "Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception". In it, it showed really great success rate for mobile surfaces.
<img width="964" alt="image" src="https://github.com/heyitsaamir/blog-next-js/assets/48929123/9d010eb7-ce02-4d7c-b0aa-26da95256218">
Here, CR refers to completion rate, which is really phenomenal. The idea is that the lack of thumb real estate forces websites to be thoughtful about what's on the screen. Additionally, [this](https://twitter.com/GregKamradt/status/1773386891368521806) tweet from Greg Kamradt further enforces the notion that the performance of the model degrades the more text that is present in the image.

To mimic its success, I decided to reduce the size of the browser such that the app would go into mobile-responsive view. The results were rather surprising. 
For my use case (add a task to the task list), the model often got confused:
| Desktop | Mobile |
|----------|-------|
| <img width="399" alt="image" src="https://github.com/heyitsaamir/blog-next-js/assets/48929123/6610a9d3-c497-4bcb-aa79-bfa655088ff3"> | <img width="414" alt="image" src="https://github.com/heyitsaamir/blog-next-js/assets/48929123/65e4b5d6-fc58-4dbb-b392-3dc518425288"> |
| The desktop version has explicit text in buttons that helps the model understand what the button does | The mobile version relies on icons more. As humans, we seem to have learned what these buttons mean (send, accept, confirm), but the model was not too sure. |

The agent would often get confused on what button to press once it had filled out the task name. It would often click the cancel button and then keep trying since the `complettion_condition` wasn't met.

### Introducing `hints`

I considered accessibility of the website in situations like this. Readers who suffer from blindness can still often use these websites even though they don't know exactly what icon or color the buttons have. They do this through HTML roles and aria-attributes. I decided to use the same principal when building the prompt for the agent. In addition to the screenshot, I would also give it some helpful hints on what elements were present on the screen. I could have sent it the dom nodes themselves, but to save on tokens and reduce noise, I wrote a custom function to give it some useful hints for all interactable components. Here is a sample of what that looks like:
```
A: type="button" text="Open/close sidebar"
SA: type="button" text="View options menu"
C: type="button" text="Expand"
D: type="button" text="Reschedule"
SD: type="a" text="Apr 1 ‧ Today ‧ Monday"
E: type="div" text="Mow the lawn
Inbox"
F: type="button" text="Mark task as complete"
SF: type="a" text="Inbox"
G: type="button" text="Edit"
H: type="button" text="More task actions"
J: type="div" text="Task name"
SJ: type="div" text="Description"
K: type="div" text="Set due date"
SK: type="button" text="Remove due date"
L: type="div" text="Set priority"
SL: type="div" text="Add reminders"
M: type="div" text="More actions"
P: type="button" text="Select a project"
SS: type="button" text="Cancel"
W: type="button" text="Add task"
```

This dramatically improved its accuracy. The model knows exactly what the buttons may mean depending on its various attributes even though it's not visually defined.


## Unpredictability of websites

Websites are unpredictable. They are built for humans in mind and often, they are built for being more visually appealing vs accessible. Because of this, developers often override best practices like rebuilding UI controls (I saw a select box being rebuilt with just divs). 

All this basically results in websites not being friendly for automation. The browser agent I built was using playwright which is a testing tool. Playwright encourages developers to build correctly or it won't be friendly when testing. However, when the tables are flipped and it's used for controlling itself, sometimes its APIs are too cumbersome for complex unfriendly websites to be built generically.

## Non determinism of the model

The AI agents are also not deterministic. You might get the same, expected, result 10 times in a row, but the 11th time it fails. [Here](https://152334h.github.io/blog/non-determinism-in-gpt-4/) is a super interesting article that talks about how non-determinism in GPT4 can be caused by Sparse MoE:
> Under capacity constraints, all Sparse MoE approaches route tokens in groups of a fixed size and enforce (or encourage) balance within the group. **When groups contain tokens from different sequences or inputs, these tokens often compete against each other for available spots in expert buffers**. As a consequence, the model is no longer deterministic at the sequence-level, but only at the batch-level, as some input sequences may affect the final prediction for other inputs

This was super unexpected to me. This means that depending on your luck and the traffic that particular GPU is experiencing, you may or may not get the result you expect. The linked article is super interesting and sheds light on a lot of non-determinism issues I had while building this. Naturally, the more investments in this technology, this is surely to get better, but for now it cannot be highly reliable.

# Conclusion

These are my latest discoveries as I build this thing and experiment more and more with this technology. I'll probably do another post with more details in the future.
For anyone interested in checking out the code, checkout the [repo](https://github.com/heyitsaamir/FIDO) here. I'd love to get more thoughts on what works for people and what doesn't and if there are any ideas on improving this tech.

Building an AI based browsing agent

Javascript is inherently not a type-safe language. You don't get compile-time errors when you are doing potentially unsafe operations between incompatible types. Typescript has become a defacto way of improving javascript's shortcomings when it comes to type-safety and has actually made javascript a viable language to build large scale applications.

Let's take an example:

```ts
const foo = 5;
const bar = 'five';

...

if (foo !== bar) {
	console.log('foo is not bar');
} else {
	console.log('foo is bar');
}
```

With pure javascript, this piece of code compiles, and logs `foo is not bar` as expected. However, in any non trivial piece of code, the compiler should tell us that this operation is moot, and we might be breaking some assumptions. Let's update thie code with typescript:

```ts
const foo: number = 5;
const bar: string = 'five';

...

if (foo !== bar) {
	^^^^^^^^^^^ "This comparison appears to be unintentional because the types 'number' and 'string' have no overlap."
	console.log('foo is not bar');
} else {
	console.log('foo is bar');
}
```

Here typescript gives us a nice helpful message saying, hey you might be doing something wrong comparing two different types.
But what happens when two variables are of the same data type:

```ts
function convertToDate(isoDateString: string): Date {
	return new Date(isoString);
}
```

In this example, `convertToDate` takes in an isoString and converts it to a date object. This is an extremely common operation in javascript. Many APIs will return json results with iso dates typed as strings and it's the responsibility of the client to convert the value to a `Date` object if the client needs to use it as a date. The `convertToDate` function here takes _any_ string as an input which can lead to a false sense of security that the code we're writing is safe. Take this code for example:

```ts
const apiDateString: string = '2023-01-01T00:29:32.431Z';

let convertedDate = convertToDate(apiDateString); // okay!

...

const userInputDateString = 'January 1st, 2023';

convertedDate = convertToDate(poorlyNamedVariable); // type-safe but invalid
```

Here, our code doesn't provide any type-safety that `userInputDateString` is in fact an isoDate. It's a string so javascript and typescript are both happy. As you can imagine in a large scale application, if a developer is unaware of how `convertToDate` works, they may feel the need to perform validation before calling the function to be sure that they are supplying the right type of input. Or if the function throws, then they would need to handle a validation error everywhere that this function is called.

Is it possible for us to differentiate the types of `apiDateString` and `userInputDateString` such that we get type-safety? This is where brand types come in. A brand type adds another layer of safety to distinguish vales of the same data type. If we modify the above to something like this:

```ts
function convertToDate(isoDateString: ISODateString): Date {
	return new Date(isoDateString);
}

const apiDateString = '2023-01-01T00:29:32.431Z' as ISODateString;

let convertedDate = convertToDate(apiDateString); // okay!

const userInputDateString = 'January 1st, 2023';

convertedDate = convertToDate(userInputDateString);
 // ^^^^^^^^^^^^^^^^^^^
 // Type 'string' is not assignable to type '{ _brand: "ISODateString"; }'
```

If we could create completely new type safe data types, the above would give us a compile time error. The `convertToDate` function only accepts `ISODateString` types. If you (or an api) is providing some guarantee that the value being supplied _is_ an `ISODateString` type, then the code will compile. Otherwise, it'll throw a compile-time error. This is possible with typescript with branded types. Here is what a generic `Brand` type looks like for string types:

```ts
export type BrandedString<TName extends string> = string & { _brand: TName };
```

For an ISODateString, you may get:

```ts
export type ISODateString = BrandedString<'ISODateString'>;
```

If a variable is assigned to an `ISODateString` type then, it's safe to be sent to `convertToDate`. Of course, you can use [narrowing and type-predicates](https://www.typescriptlang.org/docs/handbook/2/narrowing.html#using-type-predicates) to perform validation from a regular string to assert that the string _is_ in fact a branded string. Once that validation is done, however, we can be confident that the value being passed around is a true ISODateString.

```ts
const isISODateString = (input: string): input is ISODateString {
 // Really naive way to do validation, lol
	return !isNaN((new Date(input)).getTime());
}

if (isISODateString(userInputDateString)) {
	const date = convertToDate(userInputDateString) // okay!
}
```

We use this technique across the codebase at my company. We use it for:

1. entity ids (there is no point in comparing `UserId` and `CompanyId` since they are always a disjointed set).
2. data that has a unique format for data transmission (like date strings)
3. metrics (weight metrics should never really be compared against spatial metrics)

In fact, as I was building out the [bookmarks](http://www.aamirj.com/bookmarks) page, the API (I use [Raindrop.Io](https://developer.raindrop.io/)) returned a format that resembled fields that contained some of the above types of data. I had to build a type for the json anyway, so my type looks something like this:

```ts
export type BookmarkId = BrandedNumber<'BookmarkId'>
export type ISODateString = BrandedString<'ISODateString'>;
export type Url = BrandedString<'Url'>;

export interface Bookmark {
	note: string
	_id: BookmarkId
	title: string
	link: Url
	created: ISODate
	lastUpdate: ISODate
}
```

Now we can have additional typesafety with the `created`, `lastUpdate`, `link` and `_id` fields. Here is a typescript [playground](https://www.typescriptlang.org/play?ssl=25&ssc=2&pln=19&pc=1#code/KYDwDg9gTgLgBDAnmYcBCUCGA7AJsXAZRigEtsBzAHgBUA5TAW1VBmDwGc4OTyKA+OAF5uvSnABkcAN5wA+gCMseAFxx6TVAF8A3ACg9oSLATJUASUIB5ACKY2xMuJEYc+ImOoByS7fvBHUghKL359PQAzAFdsAGMYIOw4WOCAN2BYGgg7NgAKUg5s-0DKNV8cgM8ASjUKmT0ASChgGCioJOxgAHc4CvzCipKKKv0tAxTsHjhMMFJBz2E4LwAmAAZlgGYAWlWARh3dmlXVlWWAThUN5YA6ABYN3YAtL2mucuLPcIAbFuS0jLYuDqIgm6UyRTyMzmHycwx0cAA9Ai4BAANaYRAAQnGwSmUQ4GXM2DAURg81hiy8ACkcFFMFBEHBdjwADRwNabLzhUEAgjAuB-bBgmBZPr4wnE0nkvgjPQC+UKxVK5WI5EAPQ1mq12q1cpV+oNSPUZiWPFhLwKcGwEHgmA4HFIFGwmAUPwQEFMKCWskUylwagARO8HLxghQA-CtF4cZN4AVg5UKSJ8pKYGozTK1OQSXG3tZpeJpI1mq12nACgw6Llcp0en0vHbcBFGxEvFUqtcKC0aKRmLl26MDKQInB+gmhrlxVAiTmC8MqvUGhMprh-IseeCxQTp6m5wujWiMditEA) with the above concepts.

Using Branded types in Typescript


# Signs of addiction

1. Constantly reaching into my pocket for my phone, just to see if _something_ happened that needs my attention.
2. Deep feeling of discomfort when I'm away from my phone.
3. LIttle to _no_ time without access to my phone

# Reasons to break this addiction

- I feel like I have lost the ability to be comfortable with boredom. I am _never_ bored. Studies show that boredom is actually good for the creative side of your brain. That's where ideas thrive, connections are formed, and you have the ability to think deeply about any problem. Funnily enough, the only place where I don't have access to my phone is in the shower, and I am aware of all the ideas that I come up with there.
- I have lost the ability to focus on a problem for too long. When my phone is around, any idle moment, or any dip in stimulation for my brain hurts. And, I've been trained to fill that dip with a look at my smartphone. I feel the effects of my ability to not being able to focus in various parts of my life.
- Need to problem solve things _immediately_. I am uncomfortable with with things not being perfect. My level of anxiety increases when my life is not at some stable level of equilibrium. I believe that being so attached to my phone, I've been condition to solve any small problem _immediately_, without being okay with some level of discomfort. This has also reduced my ability to problem solve I think.
- Memory - These days, I feel like it takes a lot more to commit things to my long term memory. I actually feel like I forget things much faster. This constant context-switching definitely feels like it is contributing to this problem. I feel the same way when I have Slack opened. The number of times where I'm doing something and I have to pause and think "wait, what I'm doing here again?" feels like it's increased over the past few years.
- Procrastination - My phone doesn't cause me to procrastinate _too_ much. I'm someone who doesn't have a lot of social media presence. I'm on LinkedIn and Twitter these days only. But Twitter can sometimes take me down a path where I scroll for far too long without getting too much in return. It feels like junk food, but for my brain.

# What is the main problem?

- The phone is too interesting - social media, email, slack, etc are too easily accessible, and they give that dopamine hit that stimulates some innate part of our brains.
- A phone is both a place of productivity as well as a place of procrastination - this makes it hard to put a hard limit on phone usage. Sometimes, I need my phone for weather, calculator, maps or music.
- The need to be near it in case Bushra, or some family member needs me. It makes me feel like I can't leave my house without my phone. Or I can't keep my phone in another room for a few hours.
- Points from Matt D'Avella's [video](https://www.youtube.com/watch?v=cV5GWt26qDo&t=11s)
  - Rest _is_ productivity - Like actual, restorative rest. Being on your phone, and being stimulated _is not_ rest. People are busy - with work, kids, relationships - but people feel even more busy because they are not getting restorative rest between those busy things.
  - There is this massive obsession with multitasking - what does it mean to sit and have coffee, and _just_ have coffee. What does it mean to drive, and _just_ drive.
  - Using your phone is a twitch, that makes any awkward situation comfortable. Standing in line, or before sleeping (idling), using your phone fills that time which makes those moments _comfortable_, but they are taking away from important moments in your life that are used for rest.

# What am I trying?

- No social media at all on the phone - getting rid of all scrollers. These apps are designed to keep you on the platform because their success is evaluated by DAU, and hours spent on them (to eventually get you to view ads)
- Being aware of the twitch (the need to fill the gap with a look to the smartphone)
- No notifications? Have not yet started this one.


Addiction to my Phone


I'm going to be a dad in a few months. It's scary and exciting at the same time. When we found out (in July), it was definitely far more scary than exciting. But I think the ratio has settled out a bit and we are running a relatively even dose of both right now.

Bushra has been dealing with a myriad of physical changes (from nausea in her first trimester, to discomfort sleeping, and then ofcourse now, something inside her kicking her softly as a gentle but constant reminder). I have the luxury of not having that so I am reminded of this big life changing event happening through other means:

1. Stroller shopping - do you know how many different types of strollers there are? Each company has its own twist on this piece of equipment trying to differentiate itself from the competition in some minute way. I wish there was just one choice.
2. Reading books - I enjoy diving into new projects and this project is going to be a long one. I've started reading books on parenting ([[How to Raise an Adult]]) and sleep ([[Healthy Sleep Habits, Healthy Child]]). I'm fully aware that I'll mostly forget lessons from these books when we're in the midst of the chaos, but I like to think that I may retain some general view on these topics.
3. Setting up the nursery - this has been mostly Bushra since organization is her forte, and she really likes doing it, but I've been providing ideas and helping with building the crib or painting too

These tasks have been getting me excited to meet the baby. But there are definitely quite a few things that make me nervous.

1. I've never even had a pet. How am I going to keep this thing alive and well?
2. The way that I am, and the ideas that I have of the world, are they okay to be transmitted and taught to someone else? I know each kid is their own person, but the baseline starts from their parents and the upbringing at home.
3. Money - Shit everything costs so damn much. And Bushra may take a leave of absence from work for a year. Money will be tight....
4. Free time - I'm excited to play and raise this baby, but selfishly, I'm scared at what is going to happen to my passions and interests. Are those all going to be forgotten?
5. Marriage - Bushra and I have been with each other for a long time, and we are both the most important people in each other's lives. I don't know how the introduction of this third person in our family is going to change our dynamic.

I know our lives will change after March, but I'm optimistic that even with the chaos, it'll be for the better.

- I'm optimistic that it will make me a better human, with the added consequence that my actions will be examples to the child
- I'm optimistic that it will improve our marriage because we are going to need to work together as a team to do this whole parenting thing
- I'm optimistic that we'll be okay. The world does this on a daily basis. I just hope we can do a reasonably good job 🤞🏼


On becoming a Dad


I absolutely hated running. Until one of my coworkers (thanks Jasper!) taught me that to enjoy running is to appreciate the journey more than the end. What most people do is just focus on getting there - finishing the run, chasing the feeling after the run, checking it off their to-do list. But to actually get good at running, or to even enjoy it, you need to completely ignore the end. What matters is running itself.

What this means is to forget about things like pace, and distance. Just care about _time_. How can you run for X minutes straight. Then overtime, and if you care for it, how can you run for X + Y minutes straight. For most of us it's through running slowly. For most of us this means running for shorter periods of time, or for running slow. No, _slooooow_.

Slow enough that you could have a conversation with a person. Slow enough that you don't need to listen to fast-paced music to get through running, but instead you are able to listen to an audiobook or a podcast. The idea is that as you go slow, you build up endurance and muscle strength at the same time. And over time, your pace, and speed all improve automatically. At least, enough to plant that seed that will make you enjoy and appreciate running.

Shockingly, this is working for me. I actually look forward to going out for runs. It's been a great incentive for me to take a break from working and go out and listen to a podcast. My runs haven't been exceptionally long (~30 mins) or fast-paced (between 11.30 - 12 minutes / mile), but I've run the whole time, and not been terribly out of breath or hating my time outside. In fact, I'm starting to actually enjoy being outside, breathing in fresh (cold) fall air and getting through podcast by podcast. Who knows, maybe I'll graduate to audiobooks soon.

It's unfortunate that my spark for running came in fall, so close to winter, when the weather is soon going to be rainy and terrible for running.

But maybe this is what I needed. To get started with this hobby. And if I remember all that I wrote in this post, I'll pick it back up easily.


Journey to enjoy running


Recently, we had to build an oauth flow for a Slack App into our company's web app. An oauth flow is a mechanism by which an app can request a user's token to a service such as Slack (or Discord, or other). This allows the app to perform useful actions on behalf of the user. A typical oauth flow looks like this:

1. An app sets an agreement with a service. It tells the service why it wants to use it, and that it's intents are not malicious. It sets the possible scopes that it might want to request from the service. In return, the service awards the app with a specific `AppSecret` (along with a public `AppToken`). Now the service knows that this app may request.
2. When setting this up with the service, you also provide the service with a redirect url that the service redirects to along with a temporary auth code. We use this code later on.
3. On the app's website, or front end, it can create a url to the OAuth entrypoint on the service's website. The url is generally built using the app's `AppToken` from step 1.
4. Once a user decides they want to enable some specific integration between the app and the service, they may click on this link to begin the OAuth flow. They are taken to the service's OAuth entry point.
5. Usually on this page, they are prompted to log into the service, and grant access to the App requesting a specific number of scopes.
6. Once granted, the service redirects back to the App along with a temporary code (as mentioned in step 2).
7. The app then makes a call to its backend (you don't want to do this part on the front end because it requires the app's secret) with the temporary code.
8. The app now needs to make a call to the service with this temporary code to ask for the actual `oauthToken` for the user. This token is the token that needs to be used to make calls to the service on the user's behalf later on. When making this call, the app also needs to supply the service with the `AppSecret`. The service can now guarantee that this verified app should have the `oauthToken` for this user.
9. The app typically stores this oath token in some secure location (a db, or a secret storage).

As you can imagine, clicking on any url in step 4 temporarily removes a user from your website to take them to the Oauth page on the service's website. This is not a great user experience because

- Your app is unmounted from the browser and remounting the app always incurs some latency.
- If this was on a screen where the user may be working on a form, they may lose all their changes
- It leads to disorientation for the user ("where was I again"?)

To combat this, a typical flow is to pop open a popup to do the oauth flow and then close the popup once the flow is completed. There are a few pieces for this to work:

1. The user clicks on the button / link, and a popup pops up that takes them to the service's oauth entry point page.

   - Note that often the user needs to allow popups for this to work.

   In my React app, this looks something like this:

   ```tsx
   const oauthLink = getOauthLink();
   const openPopup = useCallback(() => {
     if (setupLink == null) {
       return;
     }

     const popup = window.open(
       oauthLink,
       undefined,
       `height=800,width=800,resizable=yes,scrollbars=yes,toolbar=yes,menubar=no,location=no,directories=no,status=yes`
     );
   }, [onComplete, onDismiss, setupLink]);
   ```

2. Once we make this call, we are going to start listening for a particular event message on the current window. This message will be sent from our popup window to inform this window that the oauth has been completed. That code will look like this:

   ```tsx
   const OAUTH_COMPLETED_MESSAGE = 'OAUTH_COMPLETED_MESSAGE';
   const oauthLink = getOauthLink();
   const openPopup = useCallback(() => {
     if (setupLink == null) {
       return;
     }

     const origin = window.location.origin;

     const popup = window.open(
       oauthLink,
       undefined,
       `height=800,width=800,resizable=yes,scrollbars=yes,toolbar=yes,menubar=no,location=no,directories=no,status=yes`
     );
     if (popup != null) {
       addEventListener(
         'message',
         (e) => {
           if (e.origin !== origin) {
             // Other websites could send messages too, we don't care about those.
             return;
           }

           if (e.data === OAUTH_COMPLETED_MESSAGE) {
             onComplete();
           }
         },
         false
       );
     }
   }, [onComplete, onDismiss, setupLink]);
   ```

3. After the user accepts the request on the service's website, they are redirected back to the app. We are still on the popup at this point.
4. As part of the redirect url, the temporary code is also returned. Our app is loaded (in the popup) and the call is made to the app's backend to authenticate (with the temporary code), get the oauth code, and save it.
5. Once this call is made, we are going to raise an event on the source of this current window. We are also going to start listening for ack from the original window to let us know that it has received our message. Once the popup gets the ack, it'll close itself.

   ```tsx
   function afterSendingOauthTokenToServer() {
     if (window.opener == null) {
       return;
     }

     window.opener.postMessage(OAUTH_COMPLETED_MESSAGE, '*');
   }
   ```

6. Our original window that was listening for this message is going to refresh the component that might need the oauth token to function.
7. It also raises an ack for the popup.

   ```tsx
   const OAUTH_COMPELTED_MESSAGE_ACK = 'OAUTH_COMPELTED_MESSAGE_ACK';
   const openPopup = useCallback(() => {
   	  ...
       if (popup != null) {
         addEventListener(
           'message',
           (e) => {
   					...
             if (e.data === OAUTH_COMPLETED_MESSAGE) {
               onComplete();

   						e.source?.postMessage(
                 OAUTH_COMPELTED_MESSAGE_ACK
               );
             }
           },
           false
         );
       }
     }, [onComplete, onDismiss, setupLink]);
   ```

8. The popup receives the ack, and closes itself.

   ```tsx
   function afterSendingOauthTokenToServer() {
   	...
   	addEventListener(
         'message',
         (e) => {
           if (e.origin !== origin) {
             return;
           }

           if (e.data === OAUTH_COMPELTED_MESSAGE_ACK) {
             window.close();
           }
         },
         false
       );
   }
   ```

And that's it! The experience is a lot less intrusive and much cleaner!


Building an OAuth flow using a popup

My email inbox is a mess. Especially these days. Even when I unsubscribe ruthlessly, I accumulate tonnes of emails that just stick around. Amazon confirmation emails, 2fac emails, etc etc all add up and clutter up my inbox. Those emails are important so I don't want them to unsubscribe to them, but I would like to get rid of them after some time. Sometimes, even the subject is enough, but I'm too lazy to actually delete them.

So I decided to build a Gmail plugin that would help me with this problem. My requirements was pretty simple - Allow me to easily configure a way to delete some particular types of emails after some period of time.

Google provides [AppScript](https://developers.google.com/apps-script) as an easy way to built automation around its products. I've briefly looked into it in the past (I built a email-calendar [plugin](https://github.com/heyitsaamir/sbp-booking-to-calendar) for my rock climbing gym appointments when you had to book slots during core-pandemic), so it was the natural choice. My previous experience was an extremely simple script, but here I needed to build some sort of UI to interact with my inbox as well.

AppScript has a concept of add-ons which show up on the right hand side of its apps. The addons are a convenient way to developers to have context on what's being displayed in Gmail and use that to interface with their apps.

# Rough plan

Here's the rough plan for me building this thing out.

1. Figure out how to get some sort of UI showing - This UI will help me mark a message type and select after how long I want it to be deleted (or archived)
2. Save all this information in a database
 - Bonus: How do you link a database with an add on?
3. Schedule some way for all the new threads that come along to be removed once they have been in the system for more than the allowed period
4. Have the UI also show me all the current rules, and also allow me to remove those rules.

# Just get something showing

I created a new add-on and wanted to get some sort of UI showing. Luckily, app-script docs are awesome, and they include code-snippets. There's one for building some UI [here](https://developers.google.com/apps-script/add-ons/gmail/extending-message-ui). Throwing that in my appscript and after having some trouble installing the add-on, I was pleasantly surprised to see it rendering some controls when I accessed via a Gmail message

![JustSomething](https://github.com/heyitsaamir/blog-next-js/assets/48929123/318a5336-e29e-4a02-ba28-a00b621e5bd1)

Hurray! Now we're cooking. We have some context from the message that's opened, and also some way to perform actions. Sweeet. The code looks something like:

```jsx
function loadAddOn(event) {
 var accessToken = event.gmail.accessToken;
 var messageId = event.gmail.messageId;
 GmailApp.setCurrentMessageAccessToken(accessToken);
 var mailMessage = GmailApp.getMessageById(messageId);
 var from = mailMessage.getFrom();

 var openDocButton = CardService.newTextButton()
 .setText('open docs')
 .setOpenLink(
 CardService.newOpenLink().setUrl(
 'https://developers.google.com/gmail/add-ons/'
 )
 );

 var card = CardService.newCardBuilder()
 .setHeader(CardService.newCardHeader().setTitle('My First Gmail Addon'))
 .addSection(
 CardService.newCardSection()
 .addWidget(
 CardService.newTextParagraph().setText('The email is from: ' + from)
 )
 .addWidget(openDocButton)
 )
 .build();

 return [card];
}
```

It uses a framework using cards using a service called [CardService](https://developers.google.com/apps-script/reference/card-service/card-service?hl=en). Glancing at the API, it seems to have major UI components for our goals.

# Database

The next problem we have is that we want to store rules for messages that we want to trash or archive somewhere. This database is going to keep a record of all the rules, and also potentially be used as a way to keep track of all the threads that get scheduled for action.

Doing a quick search lead me to [JDBC](https://developers.google.com/apps-script/guides/jdbc) which is a compliance for database types (databases include mysql, cloud sql etc). However, it is not free to use this service for free in app-scripts:

> [Google Cloud SQL](https://developers.google.com/sql) lets you create relational databases that live
> in Google's cloud. Note that, unlike Apps Script,
> [Cloud SQL is not free](https://developers.google.com/sql/pricing).

I obviously don't want to spend any money on this side project. My next thought was to just use a Google sheet as a database. I don't especially care about speed, and google sheets are easy to access and modify if needed. Appscript has a [Google Sheets api](https://developers.google.com/apps-script/reference/spreadsheet?hl=en). The main problem with google sheets is that it has has a cumbersome API that isn't really meant to be used a database, but we can work around it for our simple app.

I was thinking having two sheets, one for the rules (this will be set by the addon that we create), and one for all the scheduled deletions and archiving (this will be set by the scheduled task that will use the rules that were created).

For the `Metadata` sheet, we will have the following columns:

- Sender - the sender of the thread
- Matcher - the regex matcher that will match against the subject
- Take Action After - the time after the thread comes in to take the action (like 1 day, or 3 days, or a week)
- Action Type - Trash or Archive

For the `Pending Action` sheet, we will have the following columns:

- Thread Id - the id of the thread
- Action Time - the time after which the action should be taken (this will be set by the scheduler)
- Action Type - Trash or Archive

The db creation code looked something like this:

```jsx
const DB_NAME = 'AutoDelete-DB';
const METATDATA_SHEET = 'Metadata';
const PENDING_ACTIONS = 'Pending Actions';

function getDb() {
 var existingSheet = DriveApp.getFilesByName(DB_NAME);
 if (!existingSheet.hasNext()) {
 return null;
 }

 const spreadSheet = SpreadsheetApp.open(existingSheet.next());
 return {
 metadataSheet: spreadSheet.getSheetByName(METATDATA_SHEET),
 pendingDeletionSheets: spreadSheet.getSheetByName(PENDING_DELETIONS),
 };
}

function getOrCreateDb() {
 var existingDb = getDb();
 if (existingDb == null) {
 var existingSheet = DriveApp.getFilesByName(DB_NAME);
 var newSpreadsheet = SpreadsheetApp.create(DB_NAME);
 var metadataSheet = newSpreadsheet.getSheets()[0];
 metadataSheet.setName(METATDATA_SHEET);
 metadataSheet.appendRow([
 'Sender',
 'Matcher',
 'Take Action After',
 'Action Type',
 ]);

 var pendingActionsSheets = newSpreadsheet.insertSheet(PENDING_ACTIONS);
 pendingActionsSheets.appendRow(['Thread Id', 'Action Time', 'Action Type']);

 return {
 metadataSheet,
 pendingActionsSheets,
 };
 } else {
 return existingDb;
 }
}
```

It feels like building a database :). I guess databases are just rows and columns at the end of the day.

# Building the Add-On

With the above framework, it was fairly simply to put together a simple addon. Here is what it ended up looking:

<video width="694" height="400" controls>
 <source
 src="/assets/blog/2022-01-08-auto-cleanup/ScreenRecording.mov"
 type="video/mp4"
 />
</video>

Using cards, I built a basic form that takes the current mail that's opened with some other options for how long the action should be taken after and the type of action, and appends a new row into the Metadata sheet of the database.

I also kept having to go back to the sheet to make sure everything was working as expected so I also built a simple card that would read all the rows of the sheet and show it as a separate card.

# Scheduled task

Once the metadata was set, it was time to actually use it and schedule some deletions and archiving for the mail that matched. Luckily, this is a first class citizen for app-script. You can simply create the functions that you want to run regularly.

I created two scheduled tasks for my purpose. I find that have multiple simple tasks with very clear goals is easier to manage than one main task that try to do too much. In this case, it was totally possible for me to combine the tasks into 1 task.

![ScheduledTasks](https://github.com/heyitsaamir/blog-next-js/assets/48929123/eb4b55aa-9b28-41dc-8ea7-8f18228b2bd9)
_The two scheduled tasks. One is to schedule all the deletion / archives. The other is to actually excute the actions_

1. ScheduleForActionTask - This task evaluates the last messages that have come in against the metadata that we've set up. If there are any matches, it appends a row to the Pending Actions sheet with when the action should take place, and tags the thread with a "Scheduled for Deletion" or "Scheduled for Archive" tag. This way if I notice a thread that I don't want removed, I can fix the issue.
2. RemoveAllPendingThreadsTask - This task simply goes through the pending threads task, finds all the rows that are past due and either moves them to trash or archives them.

# That's it!

And that's it! Here are some screenshots of various parts of this project:

![AddonFinal](https://github.com/heyitsaamir/blog-next-js/assets/48929123/ae8622cf-7617-456c-93dd-357c1dab576c)
_The gmail add on_

![TaggedMail](https://github.com/heyitsaamir/blog-next-js/assets/48929123/6dbeabd7-5482-4282-8ca7-8d3e4c2cf759)
_The mail is tagged with "Scheduled for Deletion" or "Scheduled for Archive"_

![Logs](https://github.com/heyitsaamir/blog-next-js/assets/48929123/844f4941-0ff7-4d9c-9f1d-718f5172794e)
_The logs that are generated after each run. These are useful for me to debug issues when they arise._

Building a Scheduled Auto Deletion script

# Windowed Pagination with Relay

Relay-style pagination is a popular option for paginating graphql queries. It differs from offset-style pagination in the sense that it uses cursors to keep track of the start and end of pages as opposed to offsets. This allows you to begin pagination at a particular node as opposed to a particular offset. There are a number of tradeoffs between the two styles of pagination. More details can be found [here](https://dev.to/mandiwise/graphql-pagination-primer-offset-vs-cursor-vs-relay-style-pagination-1a60).

As mentioned in the article above, relay style pagination is uni-directional by design. This means that it's especially apt to infinite-scroll UI, but not so much for classical windowed-style pagination. That's not to say that that's not possible.

## Problem

Our systems were initially built with infinite-scroll, but over time we realized that infinite scroll did not make sense for our systems. Infinite scroll UI is more suitable for data that's changing quickly, and less suitable for tools where you may want to pick-up where you left off, and have the page be bookmarkable. Additionally, we are not a social media app, and we are not optimizing our data for discoverability, but rather for organization (sorting, filtering, etc). For this, we wanted to move toward a window-ed paginated approach.

## New UI

This is what our new UI looks like.

```jsx
1 of 20 of 1000 | 'Prev' < > 'Next'
```

The components we need to get this to work are on top of relay's spec

- total count
- current page offset

Relay-style spec states that the graphql response looks like:

```jsx
fragment MyConnection on Connection {
 edges {
 cursor
 node {
 ...NodeDetails
 }
 }
 pageInfo {
 hasPrevious
 hasNext
 startCursor
 endCursor
 }
}
```

Additionally, `hasPrevious` can be set to `false` if `first` or `after` parameters are set if it's too expensive to calculate. (Inversely, `hasNext` can be set to `false` if `last` or `before` parameters are set if it's too expensive to calculate). Formally, this looks like the following ([verbatim from the spec](https://relay.dev/graphql/connections.htm#sec-undefined.PageInfo.Fields)):

> `hasPreviousPage` is used to indicate whether more edges
> exist prior to the set defined by the clients arguments. If the client
> is paginating with `last`/`before`, then the server must return true if prior edges exist, otherwise false. If the client is paginating with `first`/`after`, then the client may return true if edges prior to `after` exist, if it can do so efficiently, otherwise may return false.

Our existing systems implemented infinite-scroll so the relay pagination and so they only had `hasNext`field implemented. The values for `hasPrevious` was stubbed to `false`since it wasn't used.

## Schema changes

To implement the new style of pagination we'll need to modify our connection schema. Relay spec says that we are allowed to modify the connection schema as long as they have `edges` and `pageInfo`:

> Connection types must have fields named `edges` and `pageInfo`. They may have additional fields related to the connection, as the schema designer sees fit.

Here are the changes we are going to implement to get the desired result:

```jsx
fragment MyConnection on Connection {
 edges {
 cursor
 node {
 ...NodeDetails
 }
 }
 pageInfo {
 hasPrevious
 hasNext
 startCursor
 endCursor
 }
 count
 edgeRange {
 start
 end
 }
}
```

Other options I had considered was adding an `index` field to edges (hard to get this value because it's not trivial to compute)

We are going to add fields `count`which will be the total count of all the nodes in the query, and `offset` which is the count of nodes after the page that's requested. These two fields also allow us to more accurately calculate `hasPrevious` (for forward pagination) and `hasNext` (for backward pagination). Here's how we'll calculate this value:

`hasPrevious` = `offset` > 0

The relay connection code will look something like this:

```tsx
type RelayConnection<Node, Cursor> = {
 edges: Node[];
 pageInfo: {
 startCursor: Cursor | null;
 endCursor: Cursor | null;
 hasNext: boolean;
 hasPrevious: boolean;
 }
 totalCount: number;
 edgeRange?: {
		start: number;
		end: number;
	};
}

const getRelayConnection<Node, Cursor> = async ({
 first,
 after,
 before,
 last
}: args): RelayConnection<Node, Cursor> => {
 // Direction here is 'forward' or 'backward'
	const paginationArgs = getPaginationArgs(after, before, first, last);
 let edges = paginator.edges(paginationArgs);
 const totalCount = paginator.count(paginationArgs);
 const remainingCount = paginator.countRemaining(paginationArgs);

 const { direction } = paginationArgs;

 const hasNext = direction === 'forward' ? remainingCount > edges.length : remainingCount < totalCount;
 const hasPrevious = direction === 'backward' ? remainingCount > edges.length : remainingCount < totalCount;

 const startOfPage = direction === 'forward' ? totalCount - remainingCount : remainingCount - edges.length;
 const endOfPage = (direction === 'forward' ? startOfPage + edges.length : remainingCount) - 1;

 return {
 edges,
 pageInfo: {
 hasNext,
 hasPrevious,
 startCursor: edges[0]?.cursor,
 endCursor: edges[edges.length - 1]?.cursor
 },
 count: totalCount,
 edgeRange: {
 start: startOfPage,
 end: endOfPage
 },
 }
}
```

[Here](https://infinite-to-windowed-pagination.netlify.app/)'s an example that implements both inifinite scroll pagination and windowed pagination with relay using this method. Source code can be found [here](https://github.com/heyitsaamir/InfiniteToWindowedPagination).

aamir j.