Networking on Sidequesting

Agentic Workflow as of October 2026

Sun, 18 Oct 2026 11:32:06 -0400

tl;dr:

This post describes my first current very rough approach to agentic development. This approach is heavy on docs, folder structure, and copious tools and scripts for verification feedback.

My current approach is codified by the Winetech Standard Repo, which is a barebones skeleton of the structure and built-in agent ready documentation to get started on a new project.

Actual Take-Aways

Use ADRs
Prune memories, AGENTS.md, and other context-filling documents regularly
Provide a space to put observations found during their tasks so don’t get distracted and information doesn’t get lost
Keep as much inside the repo as possible, to provide agents with the context on demand that they need
Provide tools and scripts that can give negative feedback to the agents so they can quickly course-correct on their own
Give them a runtime of your application to test against in a live setting. Static analysis is not enough.

Table of Contents

Prerequisites

People get different results from the different workflows, tools, models and harnesses they use, so I’d like to set out what my particular set up looks like at the outset, so you can adjust your own expectations.

Models

Big Models
- (Primary) Claude Opus 5.5 (xhigh)
- (Secondary) GPT Sol 6.1 (xhigh)
Fast
- (Primary) Claude Sonnet 5.5 (medium)
- (Secondary) GPT Terra 6.1 (medium)

I have previously experimented with Mistral, and have only heard good things but not yet tried DeepSeek, GLM, or Kimi.

Harnesses

Claude Code
OhMyPi (omp).

Be aware that I live 90% of the time in VSCode, so I also use the the Claude Code Visual Studio Extenstion which I like because I can have rich markdown and code editing support side by side with the LLM output, like this:

However, I also like omp because it has Language Server support built in, so it can do really efficient and thorough symbol renames, etc.

Because Anthropic doesn’t permit using their Max subscriptions in other harnesses, omp only get used when I need to use OpenAI’s models.

Subscriptions

Anthropic Teams Max Subscription
OpenAI GPT pay-as-you-go pricing

Because of this set up, I prefer Claude because my usage is much less restricted. However, due to Anthropic’s removal of some “bonus tokens” they’ve been giving out for a while, I am now routinely getting rate limited around Thursday, so it’s become common for me to end my week in omp with GPT Sol.

Project Ownership

I largely possess autonomous control over how I approach my repositories, so I can structure them however I choose. The descriptions below may not be feasible in your own setups, due to team size, code review requirements, managerial processes, etc. But maybe some of the things herein can still be useful.

Folder Structure

LLMs love to have context, and to have on-demand access to even more context, so I have a folder structure with some initial files that supports their access patterns, and it has been working for me so far.

I won’t explain it in full here, but you should check out my Winetech Standard Repo, which lays out much of how I’ve started to structure my repositories these days.

In brief, it takes the following shape:

.
├── docs
│   ├── repo
│   │   ├── adr
│   │   │   └── 0000-adr-format.md
│   │   ├── investigations
│   │   │   └── README.md
│   │   ├── misc
│   │   │   ├── architecture.md
│   │   │   ├── debugging.md
│   │   │   ├── history.md
│   │   │   └── README.md
│   │   ├── plans
│   │   │   ├── inprogress
│   │   │   ├── README.md
│   │   │   └── unimplemented
│   │   │   └── backlog.md
│   │   └── workflows
│   │   └── README.md
│   ├── tickets
│   │   └── README.md
│   └── vendored
│   └── README.md
├── README.md
└── repotools
 ├── public
 │   ├── go
 │   ├── python
 │   └── rust
 └── README.md

Each folder gets a README.md explaining its purpose, and what it’s subfolders are for. For example, here is the main docs/README.md file, abbreviated for including in this post:

# Docs

The front door to a project's documentation. It says what lives where and points to the separate rules file and the scripts README.
Vendored documentation is available in `vendored/`, so that agents don't have to go to the Web to learn about the details of their dependencies.

## Main documents

Standalone reference documents, listed in a table: a map of the system as it is, a symptom-first debugging index, a history of what was removed and why, rules for code that must survive a special compile step, a design and style guide, and an end-user guide that may lag the code.

## Workflows

Step-by-step procedures, one file per recurring task. They cover building and testing, running and driving the app, reading logs, releasing, and querying a local backend. They also cover guides for risky kinds of change and one page per automated check script.

## Architecture decision records

A numbered log of significant choices currently in force, each with its context, alternatives and consequences. Records are edited, replaced or deleted, never marked "superseded". Reversed designs go in the history document.

## Plans

Work in flight, written when work starts and deleted when it ends. One standing backlog collects problems noticed in passing, so they get recorded rather than fixed unprompted.

## Investigations

One folder per problem whose cause is still unknown. It holds the symptom, dated observations, open and ruled-out causes, and next steps. It is deleted in the same commit as the fix.

## Tickets

Defects and requests for code owned by other teams, filed one per file by owning component. Each ticket carries a priority (High, Medium or Low), a category, a validity note and verification evidence, and checks must not damage shared test environments.

Each header is important in its own right, but I want to note that I have gotten a disproportionate amount of value from the following items in particular:

`adr/`

I took this from a Hacker News post a few months ago, and although I’m using the traipsings of Architecture Decision Records, I am using my own, perhaps cursed, flavor of them. I have dropped the status portion, because in my world, they’re always accepted, and my standing rule is that they are always edited-in-place, never obsoleted. If an ADR exists in the repo, it is an active document.

I use them to record bite-sized facts about the repo in question. Here are a sample of ADRs in a few projects I am working with right now:

These are from a C# desktop application:

0002 — Layered MVVM application architecture
0005 — Use NotifyingObject for change notification
0024 — A message is a copyable dialog or a toast, and every message is kept

This is an AWS deployment:

0001 — OpenTofu for Infrastructure Provisioning
0008 — Layered Internal PKI: Offline Root + Online step-ca Intermediate + Short-Lived Auto-Renewed Certs
0012 — OpenVPN, IPv6-Only Split-Tunnel, Certificate-Only, tls-crypt-v2 Hardened

and it goes on like this. You can reference my standard ADR baseline document.

Because these ADRs are routinely referenced by the models, I endeavor to keep irrelevant information, like history, out of them. History is still important to maintain, so it goes into its own dedicated history.md file.

The models really do reference ADRs all the time, and its been such a superpower that keeps them on track and consistent. It also catches inconsistencies in my requests, where the models will tell me that my own decision contradicts ADR #XYZ, which is a good moment for me to reflect on what I really want from a given session.

ADRs are not considered static – I edit them in place, and occasionally delete, reorganize, or adjust them as needed.

`investigations/`

Sometimes a problem is found during a session, like an intermittent crash. It is helpful to provide the agent a place to put repeated observations of an issue along with a summary of the issue in the investigations README file. Once enough observations have been logged, it becomes its own dedicated session and can be used as the starting point for a full-on investigation. The benefit of doing it this way is that the agent won’t get distracted during a session, can recognize known failures, and eventually can work on this one problem full-time.

here are some example ongoing investigations I have open in one of my desktop applications:

./
├─── action-editor-hangs-on-load-and-selection
│ README.md
│
├─── studio-fails-fast-returning-to-actions-list
│ README.md
│
└─── studio-process-outlives-its-window
 README.md

And a brief excerpt of what one such README looks like:

# Studio's process can outlive its window and hold the layout

**Status: open.** Seven sightings of a stuck exit or windowless process between 2026-10-03 and 2026-10-18, and four timed closes on 2026-10-07 that did not reproduce it. The cause is not known. The leading candidate is a clipboard flush at exit, and it has not been tested. A fifth sighting, on 2026-10-09, was a start that failed and never exited, which has a known cause of its own.

## The symptom

A Studio the user has closed stays alive with no window anyone knows of and one thread left, waiting in the kernel (state Wait, reason UserRequest). While it lives it holds the layout: refreshing it fails on `gs.dll` (robocopy exit 10), and `Add-AppxPackage -Register` fails with `0x80073D02`. Nothing on screen says why.

## Observations

| Sighting | Started | Remote Desktop | Copy before close | Threads, handles, memory | Exit code | Dump |
| -------- | ---------------------------- | -------------- | ----------------- | ------------------------ | --------- | ---- |
| 1 | 2026-10-03, how not recorded | not recorded | not recorded | 1, 918, 237 MB | not read | none |

### Sighting 1 -- 2026-10-03: alive eight minutes after its close

A Studio the user closed stayed alive for more than eight minutes.

## Candidates

### Ruled out
[...]

### Open
[...]

`tickets/`

Tickets are where defects, feature requests, and larger behavioral changes go. I have outstanding instructions for agents to create tickets for items they’ve found during their sessions into this folder. The subfolder structure is less defined, but I currently have them organized by owner. For example, big items that are found in this repository are placed into $projectName/, while defects found in, for example, the backend repo thats owned by another team, goes in dependencyName/.

For example:

.
├── desktop_app/
│   ├── camera-reset-keeps-selected-profile.md
│   └── maintenance-max-log-days-drifts-on-disk.md

├── upstream_dep1/
│   └── trigger-condition-schema-disagrees-with-the-hub.md
└── README.md

The structure of my tickets here are also fluid, but I’m currently using this format for a ticket:

Reset leaves one setting in place

## Assessment

- **Priority:** Medium
- **Category:** Correctness
- **Validity:** Confirmed
- **Assessed:** <date>
- **Basis:** Reset promises the default state but keeps a manually chosen setting, so a bad choice can't be undone by resetting. Picking "Auto" explicitly still works as a workaround.
- **Verification:** Confirmed by reading the source: the reset path applies defaults to every field except this one, and the code that reads the setting still honors any stored value. The API docs describe reset as a full restore with no exceptions. Not reproduced live, since no test device was in the required state.


what the upstream system does:
 Reset restores most fields to their defaults but skips one. A device pinned
 to a manual setting comes back from a reset still pinned, while every other
 field in the response is a default.

what we need:
 Reset should clear that setting too, so the result matches a fresh
 registration. If that isn't wanted, the docs should say it's deliberately
 excluded.

impact:
 A device pinned to a setting that no longer exists can't be recovered by
 resetting it. It keeps failing, and the reset control reports success
 without touching the field that matters.

evidence:
 The default-setting function covers six fields and omits this one. It's
 called from the reset handler. Other device types have no equivalent
 setting, so only this type is affected.

workaround on our side:
 None. The UI redraws from the reset response, so it shows the setting that
 survived instead of hiding it. Users can still select Auto and save, which
 does clear it.

once fixed, on our side:
 Nothing. The UI already renders whatever the response carries, so it will
 show Auto on its own.

The categories I’m using belong to the set of

[Usability, Security, Correctness, Misc, Feature].

Occasionally I will request the agents go through each filed ticket and re-assess whether it’s still true or not. The purpose of this ticket structure is that it outlines clearly why this is an issue, and what this current repo is doing to work around it.

`backlog.md`

One item I’ve gotten a disproportionate amount of value from is the backlog.md file, which is the scratch space I provide to agents for them to put down small bugs, discrepencies, or things that need adjustment. This is similar to tickets, and indeed, until I decided that the tickets folder should also include the repo itself, is where feature requests and large changes went into. But after separating the two, this file is still very useful.

When an agent is working, it will often encounter problems. Depending on the agent, it may get distracted trying to solve them, or it may just forget about the issue by the end of the session. To make sure that this information isn’t lost, they have standing orders to note any such defects into the backlog.md file immediately, and then to continue the main assigned work. This keeps them on task, and ensures that I have a decent accounting of bugs in my code.

Not everything they put in is necessary, so it is worthwhile to manually prune the file from time to time, but by and large, the defects that Claude and GPT find have been worth addressing.

The format of my backlog file is like so:

# Code cleanup backlog

Things found while working on something else: dead code, latent defects, and behaviour that looks wrong but was left alone because fixing it was outside the change at hand. Nothing here is blocking. Each entry is a decision for the user, not a task already agreed.

**Add to this file rather than fixing in passing.** When a change turns up an unrelated problem - an unreachable class, a notification that reaches nobody, a leak, a binding that has never worked - write it down here and carry on with the change you were making. Say what it is, where it is, what it costs to leave, and what fixing it would change. If you do fix one, delete it from this file.


# Methodology
- Consider breaking tasks into sub-tasks
- You have a dev deployment of the application on this box and permission to use and modify it;
- [...]


# Backlog

## Release dependencies report unresolved trim and AOT analysis warnings

The 2026-10-16 x64 `just build aot` run succeeds but reports IL2104 for `WinUI.TableView`,
`Common.Logging`, `Microsoft.Data.SqlClient`, `Microsoft.Data.SqlClient.Internal.Logging`
and `System.Configuration.ConfigurationManager`, and IL3053 for `Microsoft.Data.SqlClient`.
ILC also says `Makaretu.Dns.LinuxHelper.ReuseAddresss(Socket)` always throws because
`Tmds.LibC` cannot be loaded. No runtime failure was observed from these diagnostics; leaving
them unresolved makes the Release build's analysis inconclusive for those dependency paths.
Triage would inspect the detailed ILC diagnostics, determine which paths the Windows client
can reach, and update or root the affected dependencies rather than suppressing the summaries.

[...]

I’ve found that the agents are exceptional at filling in this file, and on later dedicated passes, are equally exceptional at churning through it.

Many ADR discrepancies, usability problems, and bad bugs have been found this way, all without the agent getting distracted.

Verification and Feedback

Agents work really well when they can correct their own assumptions in real-time. They react very well to negative feedback when they try something, such as compile-time failures or other errors that show them they had something incorrect.

You should encourage this by building and supplying tools that the agent can call. This can be as simple as supplying the language server for it to see diagnostic information in, or as complex as a custom constellation of scripts that do work in advance. For instance, in a desktop application, I have a series of scripts that check for banned APIs, or that certain structures aren’t used:

.
├── check-serialization-contracts.ps1
├── check-source-rules.ps1
├── check-tooltips.ps1
└── check-window-ownership.ps1

Among many others. The tools provide output like:

2 failed:
Invalid API: src/models/ModelA.cs:76
{Binding} instead of {x:Bind}: src/views/LogView.xaml:862

which tell the model where one of its assumptions went wrong, and what it needs to correct before continuing.

Repo Tools

In addition to tools that provide feedback, it’s also useful to supply tools that give functionality that the agents keep writing for themselves. For instance, I noticed that the agents kept ad-hoc writing scripts to uninstall and reinstall an application, so instead of letting it waste tokens, and each time doing it slightly differnetly, I gave it a script that does it, consistently, every time. Now it knows to call install_newest.ps1 and it will wait for it to finish before continuing.

It pays to pay attention to what the agents are doing, to work alongside them.

AGENTS.md and Memories

A good AGENTS.md (née CLAUDE.md) file is mandatory for giving agents the initial overview and philosophy of your repository. Everyone knows this, so i won’t spend much time on it here.

AGENTS.md

In my case, my AGENTS.md contains things like:

a brief overview of what this repo hopes to accomplish
The toolchain used for building and deployment, with references to building.md for details
The style guidelines with references to style.md and adr/ for details
The instructions for code backlog, ticketing, and investigations, with references to the appropriate README.md files
Other tools available on this dev box, like whether Python, ffmpeg, curl, imagemagick, etc are available

Memories

The purpose of memories, as distinct from AGENTS.md or specific repo tooling is that it is meant to represent things that are not globally applicable.

Agents will often collect memories on their own, but sometimes its useful when you see persistent errors being made to encourage them to create memories under your guidance. Since I do a lot of work on Windows, and Agents have a not-unreasonable expectation that they’re in UNIX-land, I have make sure they have memories that relate to setting BOMs, and that the editors they often use eat backslashes and need different escaping rules.

I also use memories to give agents standing permission to do code commits, to make changes to the running development system, where to find temporary data, and what other nearby relevant repositories and documents are located that may be useful for the agent to be aware of.

These things are not global, and not applicable to other developers on this repository, so it’s not appropriate to put them into the repo-permanent AGENTS.md.

A side note: It is permissible to make tooling around these specific assumptions by using the repotools/$username/ folders, which is where developer specific tooling should go. There is no expectation that tools under your own folder are useful, or indeed even run, under any other user. Repo-wide tools should graduate into the repotools/public/ folder.

This is why I think there’s a large synergy to be had with the repo tools folder in an agentic setting.

Skills

I have not generally found skills to be very useful. I have nothing in particular to write here on the subject.

Pruning files when they get too big or outdated

Token budgets are not unlimited, and context size must be managed. To that end, one must on occasion revisiting the files you are asking the agents to work with on every request.

AGENTS.md should be kept under 10,000 tokens, and even that is pushing it. Remember that this is sent on every request. It is worth looking at what needs to be understood all the time, versus what only needs to be understood some of the time, versus what doesnt need to be understood at all.

If assumptions are outdated or have conflicting instructions, delete them. If things only apply some of the time, extract those sections to their own ADR or Workflow documents, and then put a reference in instead.

As simple as it can get, but no simpler.

The same rules apply for your memory files, which can be found under ~/.claude/projects/$project/memory, or other similar harness-dependent locations.

Condense, Extract, Delete.

Working with the Agents

(LLM, Harness) pairs have their own runtime quirks and should be worked with the way a craftsman knows how to work with a power-drill that whirls in one direction faster than the other, a hammer that isn’t perfectly flat and thus has slight directional preference, or a car that has a very delicate accelerator. The quirks are worth learning and using to your advantage. Don’t fight the agents, work with them.

Prompting Style

Because I have what I consider to be fairly well crafted initial documents and on-demand context available for the agents, I don’t put too much effort into my prompts, which are concise and minimal.

“Read this $document, and address the issues”
“Fix the bugs described by the cleanup”
“Continue working on this plan file”

Except in rare circumstances, I don’t write any additional context, because I’ve already given the context in the AGENTS.md, ADRs, and plan files.

Sometimes I have questions that require a more thorough examination before we can continue:

Examine the feasibility of adding the following feature to this application, and consider how many dependencies it would require to accomplish
We are currently doing $X in our application, but I want to move to $Y instead, write a plan file for this process

I have found it productive to speak in mechanical, precise terminology. I do not abridge my requests, use shorthand slang, or speak casually with the agents. I do my best to give them accurate and clear phrases to work with. It helps to orient their initial manifolds when you say things like “Consider ahead-of-time compilation issues”, or “profile this slowdown and find whether a different algorithm should be used for this subroutine”, instead of “why is it slow” or “u should find the bug”. LLMs are often reflections of the tone and skill of the user prompt, and if you are sloppy in your input you will find it is sloppy in its’ output.

Corrections

Sometimes the Agents are confident in their assertion that some things cannot be done. It may be the case that you know better than they do - in those situations it is important to supply them with relevant documentation, at which point they will happily change course and begin working.

Remember that they have cut-off dates for their training, so they will only recall information up to that point. Any change in status of something that happened after that date needs to be supplied to them.

Lean into their biases and expectations for tools and docs

Like said above, LLMs have quirks, biases and expectations, If you watch them long enough you’ll pick up this fact because you’ll see them say things like “That API doesn’t exist, I’ll try a different one”, or “Turns out ffmpeg isn’t found on this machine, so I’ll write a small script in python instead”.

These are opportunities to do two things:

Make your token usage more efficient
Make your tools better

Every time they backtrack on a choice is tokens that you’re spending, time you’re wasting, and context you’re eating up. Not all situations are equal, but on balance, you should consider adding what they’re expecting to your tools and your environment. They will be more competent in their tasks when they have what they want without the fuss of trying to work around it. They are also trained on those aspects, so they’ll be good at using them.

Additionally, especially when it comes to API surfaces, misaligned expectations are a solid indicator that your API design is lacking in crucial ways. If your agent is making these mistakes, other people’s agents are also making the same mistake. And human users may also be under same misapprehension. It is worth considering deeply whether the design the agents expected is actually a better design – very often, it is.

A brief methodology

I recently went through my Claude session transcripts of the past month with another agent (GPT-Luna-6.1 via omp because I was at my weekly Claude token limit at that time) with the instructions to:

“Read these session transcripts and find instances where the agent got confused, backtracked, realized a tool or program was missing, or used an API with an incorrect shape”.

The results were a useful set of places to start working some immediate remediation candidates. I was able to update a handful of checker scripts in the repo, and also file some tickets against a sister repository for improvements this way. Here are some samples of what that scan turned up, and the resolution I opted for.

1)
 - Expected: ffmpeg to be installed to change the encoding a screenshot
 - Actual: not installed, wrote a python script to adjust the encoding by hand.
> Resolution: install ffmpeg

2)
 - Expected: The same test date could be reused across pages.
 - Actual: The user clarified that Data Viewer needed September to get real data.
> Resolution: Add a script that seeds test data before certain diagnostic runs

3)
 - Expected: gsdrive can resize windows and click any named UI control.
 - Actual: It had no resize verb, resized by using pointer and click APIs manually.
> Resolution: Added a `resize` verb to the automation tooling

Hooks and notifications

Agent harnesses have started to support Hooks, which are a standard concept, just applied to this new tech. Because I do most of my work in a Windows VM through a Linux host, and a lot of these tasks are long running, I am often delayed in returning to a session to assess output or answer questions. To address that problem, I built a KDE taskbar widget that receives hooks from Claude Code. The taskbar widget lights up whenever an inner VM session is waiting for my assessment:

Parallelization

Sometimes I want to get multiple things done in one repo, and don’t want to wait for a single agent to churn through the tasks sequentially. If I have a set of tasks that I think are decent candidates for doing in parallel, as in, they touch different files and wont step on each other’s work, I open multiple panes of Claude Code and set them to their tasks. The first thing I writ in these prompts is:

You are working in parallel with other agents in the same checkout of this git repo. Coordinate with each other as you work.

I don’t know about other harnesses, but for Claude Code, this works very well. The agents will then Post and Receive messages to each other, sharing their file lists, seeing who has the next build pipeline and if they can piggyback off it, taking turns driving the application test suites, and generally doing a good job together. In extreme cases the agents will often decide to make a new git worktree and rebase their changes onto the main delegate branch.

I’ve found that I only have the mental capacity to kepe track about 4 agents at once, after that I begin to lose the ability to keep accurate tabs on their motion, and risk each agent going a bit off the rails.

Some Usage Patterns

Here are some patterns of interaction I have and employ with this generation of agents.

“Wrap up”

This seems to be a keyword that the frontier models are highly trained on. “Wrap up” means to them that they should finish their work in a good stopping place, and relinquish their turn. Eager agents that normally consume work items will pause, tidy up their solution, run one final test suite, and then leave the turn for the user. This is very useful in combination with the next item, the handoff document.

Handoff Documents

When context starts getting too packed, especially on long-running tasks that require sustained attention, I ask the agent to wrap up and produce a handoff document.

Handoff documents were a concept pioneered by the local agent harness crowd like the Pi harness, and has started to see adoption in other harnesses.

The idea is that the this agent will stop working, and a new, fresh agent will pick up the work. We don’t want to start with the main plan file, which has too much context, and has an unclear current-position. The handoff document exists in tandem with the plan file and represents the current state of the implementation, plus what a new agent needs to to know to immediately pick up the work where it was left off.

n conjunction with the plans/ folder from the above folder structure, a simple, sequential use of agents will produce a raw handoff.md file in the plan/ directory, and a complicated plan will get its own dedicated folder with its own handoff document.

handoff.md is a very temporary file, and is either constantly being rewritten depending on how extensive and token-intensive the task is, or just straight up deleted when no longer needed.

This goes hand-in-hand with the next item, “rewrite to be forward looking”.

Rewrite To Be Forward Looking

Plan files and Handoff files often get polluted, as time goes on, with historical state of the repository and the plan being implemented. Lots of:

(Done) Some Requisite Feature A
(Done) Some Requisite Feature B
Previously we had bugs in Set A, but I fixed it for and Set B is now unblocked
[,,,]

and similar things. This is a quirk of the LLMs that may useful to them in the moment, but for new agents picking up work, is entirely unnecessary. At some point in becomes the prudent choice to tell the agent to

rewrite this document to be forward looking

“forward looking” is a keyword taken from my observations of Opus. It often uses that phrase when it deletes old content, so I adopted it into my vocabulary to better trigger the behavior I’m looking for.

At the end of one of these requests, we get documents that are pruned of old detail, and only contain the necessary bits that are yet to be implemented. The agents are also smart enough to retain the items that are historical but highly relevant, like traps in the API or gaps in the machines expectations.

Of course, you can always give these documents a manual once-over with your own eyes to see if they left any details in that should be stricken.

Team A vs Team B

I have dominion over a number of repositories that are interlinked at various levels to each other, and I often have VSCode windows up and agents working on those repos simultaneously. Sometimes, features co-develop across repositories and require two agents across repos to work in tandem.

When this happens, I have Agent A in Repo 1 (1.A) work on some core feature, then ask for an Implementation Document for Agent B in Repo 2 (2.B). Then, I take that document, and give it directly to 2.B. Sometimes it becomes emergent that the feature set or details of the core project are missing something. To prevent 2.B from creating weird workarounds or producing a sub-standard end result, I ask 2.B for a document outlining its requests for the Repo 1 Team, which it will write me, and then I can hand that document back to 1.A, which will address the issues, and the process repeats.

This is a great way to get a really polished feature across two repositories simultaneously. The agents are indirectly asking each other questions, refining their assumptions, and creating features, fixing bugs, or polishing the experience.

I don’t combine the agents into one singular one because at the context sizes I am working with, it’s a lot easier and consistent to have each Repo Agent manage its own repo context. They can communicate to each other through me, and I can arbitrate their requests. Because ultimately, I am the consumer and the judge of their work product.

I find this “Team A vs Team B” document passing to be really productive.

Recent Worked Examples

I’ll describe some things I’ve had the agents do recently, beyond just the standard “implement this feature” and ‘fix this bug" requests.

Reducing Token Spend on Backtracking during Failed Test Runs

A Desktop GUI C# application has a dedicated test suite project. It’s regular output contained all the successes and failures, but the failures were printed at the top of the output, and the agents have a habit of using sed, trim, and select-last in powershell, resulting in the test failures not being visible to the agent. What often happened is that it saw a generic failure (exit code 1), then re-ran the test suite with a new command just to get the failing test.

The resolution was to reduce stdout noise and print detailed test failure output only, so the agents have visibility into the only part of the test suite that matters: whether a ran succeeded, and if not, which ones failed. As before, this saves tokens, context, and time.

Test Suite Optimization

A perfect mix of three things that make LLM use optimal:

A clear goal
- Bring the execution time of this test suite down to under 10 seconds
A way to measure
- time on the test suite executable
Knobs to change
- The test suite source code

With these three items an LLM can really go into perfect overdrive to improve an outcome.

In this specific case, I was able to work with the LLM to iteratively reduce a naively built C#/XAML test suite from 5 minutes per run, all the way down to 10 seconds. The steps in between were 1 minute, 30 seconds, 16 seconds, then 8 seconds.

The agent itself did a doggedly good job, but it was a tandem effort because I had to suggest parallelization and thread usage, eliminating old tests, and restructuring the project in ways that made it more amenable to testing.

Automation and Accessibility

WinUI3, Microsoft’s modern desktop framework, supports Automation Properties, which, if you are familiar with puppeteer or playwright, permits API control of the program. In the agentic era, a program that an agent can use, is a program that an agent can improve.

The first problem was getting this application to have automation properties in the first place. The solution of course was to supply the relevant documentation and then set the agents on their first task:

Add automation properties this application

Once that was done, an ADR was created saying that “all UI elements must have automation properties”.

And finally, with it all in place, all future bug fix or feature implementation sessions can now be driven start to finish with the Agent all on its own, since it now has access to every part of the application.

With vision capable models like the large frontier ones, an agent is now much more powerful and can make quite delicate judgements and assess its own work much more accurately, because now it can also drive the application on its own as well.

Migrating a Failing Robotics Sample From Gazebo to MuJoCo

A book I am following for robotics was built with the simulation program Gazebo, so all the examples in it are assuming and utilizing that program. However, Gazebo is widely understood to be bad, unreliable, have issues starting up, often misses messages passed between program nodes, and is generally a pain. This matches my experience. Some elder robotics I have spoken to, when hearing of my woes, told me that I should spend my time in other simulators like Isaac or MuJoCo. So I had Opus take all the book’s example and move them into MuJoCo automatically.

It took about a week 12-hour days of time for Opus to churn through all the issues, but in the end, I had a proper simulator environment that had no runtime issues at all, and was very stable.

Addressing and Naming

Sun, 05 Apr 2026 14:59:46 +0900

Naming

Everything on the network gets a specific hostname, which follows a template. The naming template you choose for your own network is likely to be different based on your own desires, but here’s the one I’ve chosen:

$TwoLetterSiteCode-$RoleTemplate-$instanceNumberOrUnit.

Two Letter Site Code

The two-letter-site-code is an abbreviation for sites that I have, or am likely to have, devices or vms located at. For devices, this is straight forward since they must physically exist somehwere. For VMs its trickier, because they could float around from place to place. For now, I’ve simply decided that VMs have a site code equal to their hypervisor host, but it’s open to modification in the future. E.g., ’s ln for London, se for Seattle, etc.

Physical

2-letter ascii code	Location
ln	London
se	Seattle

Additionally, we define two sets of pseudo-locations:

Virtual is used when something we own is hosted on a cloud provider like AWS or Digital Ocean:

Virtual

2-letter ascii code	Location
do	Digitial Ocean
as	AWS

And finally, “Special”, which is used for special case items, denotes with a z prefix. Special_

2-letter ascii code	Location
zz	Floating, of indetermine location
za	Network-Wide: We need a prefix to make a name match, but there’s only ever one of a thing. Those things get za.

These codes are self-describing as long as you have the intution for how the abbreviations were constructed. It is recommended to keep a lookup table detailing these codes somewhere.

Role Template

The Role Template is a sub-template of the naming scheme, and is used to store information such as:

the main role
its staging environment
its group count within that environment

Its of the form: $role.$env.$groupCount

role is an abbreviated string of things like. For example here is a highly abbriged list of roles I am using:

Code	Role
fw	Firewall
sw	Switch
hv	Hypervisor
srv	Generic Server
web	Generic Web Server
ca	Certificate Authority
ap	Access Point
zz	Other

Once again, the z prefix is used to indiciate special things that don’t otherwise fit.

This list is not exhaustive, and contains more specialized role names for specific web services being run, e,g.:

Code	Role
hue	Philips Hue
appletv	Apple TV
homeasistant	Home Assistant
obsidian	Obsidian Notes
etc.

env is a 1-letter code indicating its environment:

Code	Meaning
p	production
d	development
s	staging
t	testing

Your network may have more or different codes for other purposes.

In my scheme, if no env code is found, it is implicitly of code p for production.

groupCount is which group this item belongs in. For instance, two different dev teams may have the same VM running for their own purposes. In which case, Team A gets srv.d.1 and Team B gets srv.d.2 for their individual purposes.

If no group count is found, it is implicitly in group 1.

Instance Number / Unit

All items get a defualt -1 as the final portion of their hostname. A number greater than one indicates that the entre preceeding {Site-Code, RoleTemplate} has a second instance. For example, a -2 on a firewall could be a failover, -3 or -4 could be for load balancers.

However, it’s also possible to use colors: -green or -blue, which are used to inidicate co-equal items where no one unit is of greater implicit importance than the other. For example, “firewall-green” and “firewall-blue” are equivalent, and used in a prod/staging flip-flop gate where one is active currently, but may be backgrounded for the other at any point in time. These items can either be always co-equal, or take turns being the primary, but as far naming and setting up the hosts themselves, there is nothing intrinsict at setup time that would imply numeric naming. These items get the -color scheme.

Example

A fully built example hostname would be something like:

ln-fw-green, meaning: “green” Firewall 1, located in the London office, the primary/standby status of which would be be queries spearately
se-ap-3, meaning: the third Access Point in the Seattle office
mn-web.d.3-7 meaning: Montreal office; Dev Webserver third grouping, instance 7
mn-web.d.3-8 meaning: Montreal office, Dev websever third grouping, instance 8.
mn-web.d.4.7 meaning: Montreal office, Dev Weberverr, fourth grouping, instance 7

Typically, the unit number would be dependant on the grouping. Otherwise it should be taken that they are independent.

16 characters

Windows Domains havea 16-character limit for hostnames. We generally try to restrict ourselves to the same limit, but we also break this from time to time. If you intend to manage windows domains, or intend to manage things from Active Directory, you should take care to make things fit into 16 characters, or at least be sure that the items that will be inclued in AD to be under that limit.u

Static Assignments

For the most part, everything that gets a hostname also gets a static IP assignment. One way of thinking about what gets a static IP is that clients make do with DHCP but servers get a static IP. Anything that needs to be accessed by other things gets an IP. Your phone doesn’t need to be accessed except by you, and by physically operating it.

My switch needs to be located often, so it needs a static IP.

In our setup, we do not use DHCP reservations. You are free to use them, but we will just be using ip assignemnts on the devices themselves.

For example, our firewall will be given the following static IP: 172.30.0.1, and our main switching hub will be given 172.30.0.2. This list will include not just other infrastructure like our hypervisor, but also overlay things like Hue Bridges, or VMs serving important apps on our network. File storage? 10.96.16.35. Mail server? 10.96.16.27.

Phones, desktops, etc can just rely on getting a dynamic IP from the DHCP server, since they are not expected to be accessible except by phsyical operators.

ULA

Although the network will be primarily IPv4, it will also be created with an eye towards IPv6. One such IPv6-thing is the concept of a Unique Local Adddress (ULA), which is a specially derived IPv6 address with certain properties.

The exact reasons for why ULAs are interesting is beyond the scope of this post, but among other things, it means:

If in the future. networks merge together, there will be a statistically-zero chance of collision
We can encode a lot of data into the ULA that make the addresses self-describing in certain capacities

Using this IO Tool’s generator webapp, we can create some initial ULA templates for use. For my purposes, I used the main MAC addresses of my firewall’s built-in NIC for my underlay, and the MAC address of the SFP+ 520 card for the overlay, which mirrors the intended data that each card is expected to carry.

TODO ULA Encoding

Fixing missing, undefined, or incorrect validity periods for XCA issued certificates

Sat, 21 Mar 2026 18:02:36 +0900

Check your template applications.

Shot

Recently while trying to issue some certificates from XCA, I struggled with getting XCA to issue certifivates with a defind expiry datetime. No matter what I did to set the dates manually, the validity period expires at was always “undefined” aka “forever”.

Chaser

Turns out the the reaosn is because I was applying an XCA template that specifically checked [x] No Defindd Expiry Time.

The application of such a template overrides the expiry time of certificates, evem you specifically uncheck the option, and even if you specifically set an expiry time.

The UI won’t reflect this. The Certificate creation dialogs will tell you that you set an expiry time properly, but then you go and check the values of the created certificate and notice that the expiry date remains undefined.

So, double check your templates.

Home Networking 2 - Architecture

Sun, 15 Mar 2026 15:33:01 +0900

Physical and Logical Architecture

This post continues from Home Networking 1.

Brief VLAN legend

The vlans we’re using for now are established like so:

VLAN	Purpose
1	untagged
100	Infrastructure management panels
900	Inbound ISP traffic
1016	Trusted network
1032	Guest network

Logical Layout

Recall from earlier we mentioned Router On a Stick, implemented thusly:

---
title: Logical VLAN connections
---
graph LR;
sfp((ISP/WAN));
switch;
ap(access point);
hypervisor;
firewall;
sfp-->|vlan/900|switch;
subgraph vlan/all;
switch<-->|vlan100,vlan/all|firewall;
end
subgraph vlan/other;
switch<-->|vlan100,vlanall|hypervisor
end
subgraph WiFi Networks;
ap<-->|vlan/1016,
vlan/1032,
vlan/100
|switch
end

ISP modem goes into the switch, which in turn goes to the firewall. The ISP modem is an SFP unit, so it goes into one of the switches SFP+ slots. From there, a different switch-inserted SFP+ module connects to an SFP+ module on the firewall, over 10GB optical cables. The optical cables we’re using are “LC UPC Duplex”, meaning there’s one dedicated line for outgoing, and one dedicated line for incoming. Things can, in theory, go very fast on the internal network.

Once network packets pass through the firewall, they go back out into the switch to their destination vlan device.

ISP packets come in on a port marked for vlan 900. This is a standard marking for ISP / inbound-WAN traffic. Once it gets to the firewall, it will be sent back through the switch with whatever vlan is appropriate, defined by rules we will discuss later.

Physical Layout

Because our vlans are traveling over a physical switch with physical ports, we can’t just define the vlans in the opnsense firewall, we also have to define them on the switch.

On our 24 Ethernet / 4 SFP port switch, we have roughly the following layout:

Ports 1-12 (1/1 - 1/12 in FS switch parlance) (ethrenet) - untagged*, meaning that anything plugged into them is capable of being routed anywhere. This is done for management purposes, therefore anything plugged into this set of items should be for device and infrastructure management.
- * “Untagged” in a vlan-aware system implies vlan/1. Don’t ask me why.
Port 1 (1/1) - dedicated to the Managemnet interface of the Firewall. This provides us with direct access to the OPNSense control panel / CLI.
Port 24 (reserved) - for future expansion, this is marked as being a Trunk (aka, capable of carrying multiple vlans) carrying other, yet to be defined vlans, which is meant to be connected to a yet-to-be-installed second switch. This is meant to allow us to drop in another switch onto th network if we run out of ports on our current one.
Port 25 (SFP+/fiber) - the actual switch<->firewall traffic. This port is makred as being a trunk for every for every single vlan (EXCEPT vlan/100 management traffic) we define on the network. This is necessary because we want all traffic to flow through the firewall.
Port 28 (SFP+/fiber) (1/28) - vlan900 for ISP traffic, and is where the ISP’s SFP+ WAN modem unit is plugged into.

Or, in table form:

Port	VLAN	Notes
1-12	untagged	For infrastructure access
1	untagged	Firewall management
2	untagged	Hypervisor management
23	100	Access Point* traffic
24	Trunk, all	Reserved for future switch expansion
25	Trunk, all	Firewall traffic
26	Trunk, all	Hypervisor VM traffic
28	900	ISP WAN access

The other ports will be filled in later

*the Access Point requires some “““flexibility””” in our system because it only has one port, meaning we have to combine our Management and Client traffic over one port using multiple VLANs on one port. Ideally we would have had two ports, one dedicated for management, and the other for client traffic. It gets a little tricky, and we will discuss this later.

The actual physical port diagram looks like this:

---
title: Physical Port Diagram
---
graph LR;
switch;
isp((ISP/WAN));
ap(access point);
hypervisor;
firewall;
switch-->|1|firewall;
switch-->|2|hypervisor;
switch-->|24|ap;
isp-->|28|switch;
switch-->|25|firewall;
switch-->|26|hypervisor;

As you can see, the AP only having one port breaks the pattern a little bit, but we can work around it.

Underlay/Overlay

The network that we’re going to set up is going to that of an Underlay Overlay network. The underlay is where the network infrastructure will reside on its own internal RFC1918 IPv4 address space. Things on this space will have free access to both other things on the underlay, and to things on the overlay. Things on this network are variably on vlan 100, or untagged. Addresses on this this network will for things like the switch, the firewall, etc.

The overlay network is where everything that isn’t direct network infrastructure reside, such as settop boxes, iot devices, and user accessible VMs. The address space is also an RFC1918 address space, but deliberately separate from the underlay one. This overlay network is extenseively vlan’d to achieve high separation.

For our purposes, the IPs we choose will be as follows:

Network	Address Space
Underlay	172.30.0.0/16
Overlay	10.96.0.0/16

We choose a /16 CIDR so that we can make liberal use of the third ipv4 octet. Because this is going to be a relatively small network, we will end up using that third octet to denote the vlan of the network. This is not mandatory, but it has two nice side effects:

The network is easily categorizable at a glance
Networks will not have any overlap with each other in the event they need to perform cross-talk.

Of course. this setup limits us to 256 addresses per vlan, but that’s fine for our purposes.

Home Networking 1 - Overview

Sun, 15 Mar 2026 10:58:49 +0900

Setup and Overview

With advice from an expert friend, combined with barely even novice level talent and understanding of networks, here’s how I’ve begun to set up a network that may be used for both learning at home, and some minor commerical endevaors. One of the outstanding goals is to reduce reliance on cloud providers and proprietary solutions as much as is feasible. That process may also be a progressive one.

All of this is thanks to one specific friend. I won’t name him here, but he knows who he is. As I am a cheapskate, if there’s any kind of even remotely industrial hardware here, it is from his personal stash of equipment. In addition to his hardware, all of this is due to his many years of hard-earned expertise as a professional in the field, which I am fortunate to learn from.

For context, remmber that I am a software developer who has successfully avoided networking since university. I don’t know nothin’ about nothin'.

Overview

This setup relies heavily on VLANs, hardware that supports VLANs, SFP+ modules, and other things that I have either only vaguely heard of, or have never heard of at all before this started.

Flow

This series of posts will vaguely follow this outline:

Overview (this post)
Physical and Virtual architecture including vlans, wiring, etc
Setting up the Access Point and WiFi
Setting up the Switch
Setting up OPNSense Firewall
Setting up our Public Key Infrastructure (PKI)
Setting up OpenVPN
Setting up Proxmox
Spinning up VMs

Hardware

Lian-Li PC-Q21
- Intel X520 DA-2 2x SFP+ 10GB Network Adapter (aka, a NIC)
- Runs the firewall
Dell T420
- Same Intel X520 DA-2 as above
- Runs the hypervisor
FS S3900 24T4S network switch
Netgear WAX610 Access Point

Some questions you may have:

Why such a heavy duty switch?

Originally, I had a TP-Link switch. That, combined with the Lian-Li via one X520 NIC were supposed to be all that was needed for networking, but due to some ISP quirks related to the modem unit, the Intel branded NICs could not recognize the modem module, and a more heavy duty switch, the FS S3900, had to be brought in to handle that modem step. Aside from some general weirdness (why is the port numbering bottom-to-top? Why can’t I upload a cert directly?) because FS the compnay is itself kind of weird, the switch is working fine. But only fine. This UIs on this thing are borderline unseable.

Why an AP and not a router with OpenWRT, which surely you already had?

Astutely observed. Sadly I fried my main router, a Nighthawk R7800 (RIP to a real one), by plugging in a 48v DC power adapter by accident, so that was the end of that. Then, a backup router my friend gave me also didn’t boot, though we are uncertain if this is another 48v error, or if it wasn’t working for other reasons. In any event, I figued that as I play pretend professional-network-engineer, I may as well lean into it and get something professional. We’ll talk more about this one later, it largely works fine, with only one-and-half issues to complain about (tl;dr only 1 port, and does not yet run OpenWRT).

For those in the know, this means that we ended up moving to a Router On a Stick architecture. I don’t know enough to say whether thats good, bad, or neutral, but apparently that’s what it is.

Wires and Module knicknacks

We’re doing things with SFPs, so we also have 4x 10GB intel-compatible SFP+ optical modules, with associated LC UPC Duplex to LC UPC Duplex, Multimode (OM4) fiber patch cables to carry data over those modules.

Plus, obviously, a ton of ethernet cables of various grades, though a move to consistent Cat6e is underway.

Software

OPNSense
- Firewall, plus all the things that would otherwise be handled by the router including VLAN assignments, DHCP, etc
- Kea DHCP
  - Bunled as part of OPNSense. Somewhat quirky, just like OPNSense, but it works.
Proxmox
- Hypervisor, like Hyper-V for those coming from the Windows World like myself.
OpenVPN
- Why not Wireguard? Because I intend to have multiple people on the network at once and wireguard is mostly single-user.

There’s a bunch of other things that will make appearnces as appropriate, like certificate management software. OpenWRT in particular was supposed to appear here until the r7800 got zapped to death. And the switches run their own OSs which is not ideal, but is basically fine.

The above 3 things are going to do the lions share of work to establish our network, its perimeters, and its rules.

Keyword jumble of tech I’ve had to become acquainted with

FreeBSD
- And also its cousins OpenBSD (max security) and NetBSD (max device compatability), and why not to use them
OPNSense
- With a heathy amount of dishing on PFsense
Proxmox
“Firewall”, as opposed to just “router”, and also not just in the sense of “blocking ports” as though on a Host machine
SFP+
Infiniband
Optical wiring
Subnets and subnet masks
VLANs
- including Trunks, Tagged and Untagged, 802.1Q
Kea
DHCP at more than just a surface level understanding of “its the ip address assigner protocol”
- including DHCP pools
tcpdump beyond just “oh yeah I’ve heard of that”
UDP beyond “its the thing thats not TCP”
“Network Adapter” as opposed to “NIC”
IPv6 at deep level
- ULA schemes
RAID
Drive mirroring
ZFS
- including zpool, datasets, and iops
Underlay / Overlay networks
PKI
- including Certificate Authorities, Sub-CAs, Extended Key Usage, Subject Alternative Name, Certificate Revocation Lists, etc
TFTP
RADIUS
Virtual IP (VIP)
S/MIME
Syslog as a format (RFC5424), as opposed to the place on unix systems where system info get dumped to
EEPROM
Serial Console despite being ethernet shaped is not ethernet
Baud rates
“Management” with respect to device administration
DynDNS
LAGG

Implementing OAuth 2.0 from the RFCs

Mon, 05 Jan 2026 22:58:37 +0900

Prologue

OAuth is a common peice of web technology infrastructure that underpins many common websites. To keep things short, its how you log into websites.

But we’re here to learn more than that. First, how much detail would you like to learn? Select a skill level, and we’ll work from there.

n.b., we will only be covering OAuth 2.1 here, since OAuth 2.0 is now deprecated. That means no Implicit Flow, and Authorization will use PKCE only. Also, we’re going to be deep-diving into a bunch of RFCs, so depending on the experience level, be prepared for some technical writing.

Skill Level -1 (just the summary)

[[Graph 1 – the drawing of Login With X]]

It’s when websites or apps ask you to “Log In With Google”, “Log In With Instragram”.

Skill Level 0 (novice / just a random internet user)

[[Get lots of pictures of OAuth here]]

OAuth, specifically OAuth 2.1, though commonly just referred to as OAuth 2, is how things like “Log in with Google”, or “Log in with Instagram” work. You already have an account with Instagram, so it’s easier to just login with those credentials than to create a brand new login for yet another app.

When you select Log In With Google, you get redirected to Google’s own log-in form. This form shows you:

Who is asking for permission from you
What they want to have access to / what they want to be able to do

After considering this form, you confirm or deny the Authorization Request, and then you get redirected back to the site you were trying to log into originally. There was a bit of a bouncing from hither and tither, but it worked well and now you’re logged into the new website.

This means:

The website does not know your Google password
The website does not store your Google password
The website cannot login as you
The website cannot impersonate you
The website cannot do whatever it wants

Whats important to know here is that when you login with Google, only Google knows about your password. The website doesn’t know anything. All it does is give the website access to specific aspects your account. What, specifically, it has access to was shown to you on the Authorization Screen.

For most purposes, the only thing you need to remember is that it reduces the amount of logins you need to keep track of. Instead of creating yet another account on yet another site, you can just use your Google account.

Skill Level 1 (developer, but not too into web tech)

OAuth2 is how you login into one site with an account from another. It’s designed to avoid sharing actual credentials such as passwords, while still letting you access things associated with your account on one service, while using a different service. One example of this would be stats.fm, an app that shows listening statistics from your Spotify account. Stats.fm is not affiliated with Spotify, but it needs access to your Spotify account to work. When you log in, it asks you to “connect your Spotify account”, which brings you to the Spotify login screen. There, you log into Spotify like normal, on Spotify’s actual spotify.com domain, and then you get redirected back to stats.fm, with the redirect operation carrying some extra information. It is with this extra information that the app now has access to knowledge related to your Spotify account.

[[Animation 2 – anim including return of Access Token]]

Access Token

This is accomplished by the acquistion of an Access Token, which is string that is sent back and forth between stats.fm, and Spotify. This string is assocated with your specific account and gives its bearer access to information associated with your Spotify account. It is functionally similar to a cookie holding login information. The contents of the string are intended to be opaque to users. The contents are not important, what matters is that the string is sent over to the server exactly as-is.

Whoever bears this token has access to everything the token was given access to during the initial Authorization. Because of this property, an Access Token, and all other tokens similar to it, are known as a broad category, as Bearer Tokens. We will be referring to Bearer Tokens a lot, but just remember that a Bearer Token is a category of credential, and Access Tokens are a specific type of Bearer Token.

Access Tokens are typically short-lived. What this means in practice is that access tokens have a lifetime of about an hour (3600 seconds, this number will appear a lot as well), after which the the token becomes invalid. Continuing to use the token will result in a lot of 403 Unauthorized errors.

Refresh Token

So how do you get a new one if its expired? Does the user have to do another Authorization step? No, to get another Access Token we use something called a Refresh Token. When we got our original Access Token, a handful of other data was sent to us at the same time. One such piece data we got in addition to the Access Token was another token, called the Refresh Token. A Refresh Token’s only purpose is to be given back to the main server. When we do so, the sever sends us back another Access Token, and another Refresh Token.

[[Animation 3 – anim including return of Access And Refresh Tokens]]

Skill Level 2 (developer, in charge of integrating with an OAuth Provider)

[[Animation 4 – anim including the Auth Code, return, and exchange]]

OAuth 2 is a set of specifications for supplying third-party applications with first-party authorization (“authz”, or “what you are allowed to do”) information. What’s important to know is that [OAuth 2.1[(21 RFC)] is the current main spec, and that it defines the Authorization Flow with PKCE, and the Client Credentials flow. There’s also an extension to OAuth2 called OpenID Connect (OIDC) that deals with providing authentication (“authn”, or “who you are”) information. The difference between the two is often muddled.

OAuth 2 uses a standard set of terminology that one should be familiar with, otherwise it becomes confusing trying to understand what is talking to who and under what circumstances. The entities commonly seen in the spec are:

Authorization Server
Resource Server
Resource Owner
Resouce
Client
User
User Agent
Flow
Token

The spec defines treats these entities seperately, but oftentimes they can be co-located, making the distinction hard to grasp when thinking about it from the perspective of certain services you may have encountered in the wild.

Working Definitions

You can find the true definitions in the spec, but let’s outline them here:

Authorization Server
- This is the authority with whom you are trying offer “Log In With” capability with.
- This is Spotify, Instagram, Google, etc, where the users actual account lies
- Specifically, this is their accounts.xyz.com doman, as opposed to their generaic xyz.com domain.
- From the perspective of OAuth, the Authorization Servers only responsibility is providing access to user account checks.
- Authorization Servers hold all the Resource Owners
Resource Server
- This is the server where “resources” are.
- Typically, this is the api.xyz.com domain or the xyz.com/api/ endpoint.
- The resouce server does not Authorize,
Resource
- A resource is anything that your integrating application might be interested in
  - Liked Songs
  - Friends List
  - Calendar Events
  - Product Reviews,
  - etc
Resource Owner
- This a term describing whose things you’re interested in
- whose Liked Songs?
- whose Friends List?
- whose Calendar Events?
- From the perspctive of OAuth, the Resouce Owner is the literal user account on the Authorization Server. As a matter of philosophy, that means that a “User” and a “Resource Owner” are the same abstract entity, represented at different concrete steps of the process.
- A Resource Owner necessarily has an account on the Authorization Server
User
- The person trying to login, the person actually phsyically performing the button clicks to initiate the varius Flows
User Agent
- The user’s Browser, or the Web View used to phsyically handle the Authoriztion process
Flow
- Flows are a little more abstract, but they represent the various different types of back-and-forth to obtain Tokens
- the two major flows are Client Credentials, and Authorization
  - n.b, 1: Authorization used to be called “Authorization”, and then an extension known as “Authorization with PKCE” was introduced, and then that got absorped into the main spec, so now it’s just back to “Authorization”.
  - n.b, 2: There used to be something called the “Implicit Flow”. This is now deleted, and servers that continue to support it are out of compliance with the current specification.
Token
- A string that can be used to access the API servers on behalf of a user.
Client
- A specific application that has bene registered with the Authorization Server, and is permitted to receive and use Tokens.
- From the perspective of you, a developer integrating with an OAuth Server, this is is your application, that you have registered with the OAuth Server.

For reference, here is how the OAuth Spec depicts the main Flow

 +--------+ +---------------+
| |--(1)- Authorization Request ->| Resource |
| | | Owner |
| |<-(2)-- Authorization Grant ---| |
| | +---------------+
| |
| | +---------------+
| |--(3)-- Authorization Grant -->| Authorization |
| Client | | Server |
| |<-(4)----- Access Token -------| |
| | +---------------+
| |
| | +---------------+
| |--(5)----- Access Token ------>| Resource |
| | | Server |
| |<-(6)--- Protected Resource ---| |
+--------+ +---------------+

As comparison, here’s how Spotify depicts the same process:

These graphs are as simple as it can be, but that doesn’t make it necessarily simple.

The OAuth Dance

The OAuth Authorization Flow is like a three-person dance between the user, your Client, and the Authorization Server.

You have to craft an Authorization Request with specific parameters, then send both the user and that crafted request to the Authorization Server. At this step, your Client must wait while the two other parties dance on their own for a bit.

The Authorization Server uses that request to ask the user if they acquiesce to the request. If so, the Authorization Server gives your Client an Authorization Code, and then your Client is back in the dance.

The User is now retired from the dance completely, and its just your Client and Authorization Server now.

Using the Authorization Code, your Client hands the Authorization Server the same Authorization Code it just received, and in return the Authorization Server hands your Client back an Access Token.

At this point, the three-way dance concludes. The final result of the dance is the Acess Token, which you can now use to dance with the Resource Server, separately.

The dance is hard to understand in the abstract, and it’s much easier to understand in motion. To that effect, you can see the dance happen with real information here:

Programming Language Features I'd Like Most Languages To Have

Tue, 10 Jan 2023 12:19:34 +0900

Here are some things I am dissappointed to leave behind when I switch back and forth between languages that dont have certain features.

If Expressions

Found in: Rust; probably other functional languages

Irritatingly missing from: Everything

let x = 5 + if true {
 1
} else {
 7
}
println!("{x}"); // 6

What I have to do instead:

let x = 5;
if true {
 x += 1;
} else {
 x += 7;
}
System.Console.WriteLine($"{x}");

Ternary

Found In: Everywhere except Go

Ternaries are like If Expressions but less powerful

int i = if true ? 5 : 7;

What I have to do intead:

var i int
if true {
 i = 5
} else {
 i = 7
}

For-Else Blocks

Found in: Python

Irritatingly Missing From: Everything

def for_else(arr):
 for i in range(len(arr)):
 print(f"loop at {i}")
 else:
 print("Arr was empty")

for_else([]) # "Arr was empty"

What I have to do instead:

void for_else(arr: List<int>) {
 List<int> arr = new List<int>();
 if arr.length() == 0 {
 System.Console.WriteLine("Arr was empty");
 } else {
 for(int i = 0; i < arr.length(); i++) {
 System.Console.WriteLine($"Loop at {idx}");
 }
 }
}

Result Types

Found in: Rust; Haskell; any of the OCaml derived languages

Irritatingly Missing From: Everything

fn r(b: bool) -> Result<u32> {
 if b {
 Some(5)
 } else {
 None
 }
}

let r_t = r(true).unwrap(); // 5
let r_f = r(false); // None

What I have to do instead:

int? r(bool b) {
 if (b) {
 return 5;
 } else {
 return null;
 }
}
var r_t = r(true); // 5
var r_f = r(false); // null

Built-in Tuples

Found In: C#; Rust; Python; Most functional languages

Irritatingly Missing From: Older C#

var tup = (1,"a");
System.Console.WriteLine(tup.Item1); // "1"
System.Console.WriteLine(tup.Item2); // "a"

What I have to do instead:

var tup = new Tuple2<int, string>(1, "s");

or even worse, define my own custom tuple type:

struct Tup2<T1, T2> {
 T1 Item1;
 T2 Item2;
}
var tup = new Tup2<int, string>(1, "s");

Destructuing

Found In C#; Python; Rust; JavaScript; Most functional languages

Irritatingly Missing From: Older C#

(int, string) f() {
 return (5, "s");
}

(int t_num, string t_str) = f();

System.Console.WriteLine($"t_num: {t_num}; t_str: {t_str}"); // "t_num: 5; t_str: s"

What I have to do instead:

Tuple2<int, string> f() {
 return new Tuple2<int, string>(5, "s");
}
var tup = f();
int t_num = tup.Item1;
string t_str = tup.Item2;

Negative Array Indexes

Found In: Python; Rust (kind of)

Not Found In: Everywhere else

arr = ["a", "b", "c"]
print(arr[-1]) // "c"

What I have to do intead:

List<string> arr = new List<string>(){ "a", "b", "c" };
int len = arr.length;
int idx = len - 2;
System.Console.WriteLine($"{arr[idx]}"); // "c"

Trailing Commas for all list-like structures

Found In: Python; Go; Rust; C# (lists only)

Not Found In: C# (func parameters); JSON

C# has this for literal lists:

List<int> lst = new List<int>() {
 1,
 2,
 3,
};

While Go also has it for function parameter lists:

func f(param_a int, param_b int,) {

}

What I have to do instead: Meticulously prune commas and re-order elements whenever the list structure changes:

List<int> lst = new List<int>() {
 2,
 1,
 3
}

Numeral Separators

Found In: C#; Go; JavaScript;

Not Found In: Dart;

int i = 1_500_000;
System.Console.WriteLine($"{i}"); // 1500000

Byte Literals

Found In: C#; Go; JavaScript;

int i = 0xFF;
System.Console.WriteLine($"{i}"); // 255

Typed Range Literals

Found In: ADA

I’ve never used these because they dont exist in any language except ADA, but I really wish they were more common.

with Ada.Text_IO; use Ada.Text_IO;

procedure Learn is
 subtype Alphabet is Character range 'A' .. 'Z';

begin

 Put_Line ("Learning Ada from " & Alphabet'First & " to " & Alphabet'Last);

end Learn;

Default Function Parameters

Found In: C#; JavaScript; Python; Most languages tbh

void f(int i = 0, String a = "abc") {
 print("{i} and {a}");
}

f(); // "0 and abc"
f(5); // "5 and abc"
f(10, "xyz"); // "10 and xyz"

What I have to do instead:

void f(int i, String a) {
 ...
}
f(0, "abc");

Function Overloading

Notably missing from: Dart

string f(int i) {
 return "int";
}
string f(string s) {
 return "string";
}
string f(SomeStruct s) {
 retrun "Struct";
}

SomeStruct ss = new SomeStruct();

List<Object> l = new List<Object>() {
 5,
 "hello",
 new SomeStruct(),
};

let x = l.select(x => f(x)).ToList(); // ["int", "string", "Struct"]

What I have to do instead:

String f_int(int i) {
 return "int";
}
String f_str(String i) {
 return "string";
}
String f_class(Object i) {
 return "Object";
}

List<Object> l = [
 5,
 "hello",
 SomeStruct(),
];

var l2 = l.map((x) {
 switch (x.runtimeType) {
 case int:
 return f_int(x as int);
 case String:
 return f_str(x as String);
 default:
 return f_class(x);
 }
});

Chained Evaluators

Found In: Nowhere?


bool f(int n) {
 return 1 < x < 10;
}
f(5); // true
f(11) // false
f(0) // false

What I have to do instead:

bool f(int n) {
 return (1 < x) && (x < 10);
}

Object Assigment Shorthand

Found In: Rust; Go

struct S {
 id: i32
 name: String
}

let name: String = "xyz";
let id: i32 = 0

/// But using shorthand, as long as the symbol name and the field name are the same,
/// the field name can be omitted in favor of the symbol
let s_short: S {
 name,
 id
}

What I have to do instead:

struct S {
 id: i32
 name: String
}

let name: String = "xyz";
let id: i32 = 0

let s_long: S {
 name: name,
 id: id,
}

Spread

Found In: JavaScript; Dart;

This one has many uses, including spreading arguments into function call parameter lists but what I wnat to focus on is the ability to create new lists and objects. Taken from the MDN docs:

const parts = ['shoulders', 'knees'];
const lyrics = ['head', ...parts, 'and', 'toes'];

What I have to do instead:

const parts = ['shoulders', 'knees'];
const head = ["head"];
const lyrics = head.concat(["shoulders", "knees"]);
lyrics.push("and");
lyrics.push("toes");

Null Coalesce

Found In: C#; JavaScript;

string may_be_null(bool b) {
 if (b) {
 return null;
 } else {
 return "some value";
 }
}

var x = may_be_null(true) ?? "WasNull";
var y = may_be_null(false) ?? "WasNull";
System.Console.WriteLine(x); // "WasNull"
System.Console.WriteLine(y); // "some value"

Null Chain

Found In: C#; JavaScript;

string may_be_null(bool b) {
 if (b) {
 return null;
 } else {
 return "some value";
 }
}

may_be_null(true)?.ToUpper()?.split("_"); // null
may_be_null(false)?.ToUpper()?.split("_"); // ["SOME", "VALUE"]

If Let

Found In: Rust; Kotlin

foo?.let { baz(it) }

Instead, I have to do:

if (foo != null) {
 baz(foo);
} else {
 null;
}

Music Channel Part 3 - Branding

Wed, 27 Mar 2019 13:44:13 -0700

This is the third in a series of posts about analyzing YouTube music channels.

Part 1 - Music Channel Introduction and Data Collection
Part 2 - Analysis of Channel Activity
Part 3 - Branding

Branding

Remember that we’re basing our channel the behavioral patterns of some other channels:

Among their other patterns, we’re going use them as a base for how to brand our channel as well.

Branding in this sense is going to include:

Channel Name
Main graphics, including:
- icon
- font choices
- style, like whether it is just text or includes some form of art
Video Style, including:
- text layover
- background choice
- animations if applicable
Additional graphics, like channel banners

The types of music we’re going to upload, as well as what we’ll put in each video description, is a concern for later.

With that said, let do some research.

Majestic Casual

Name

The name doesn’t conjure up any image in specific, but does have an elevating, relaxed feeling to it. It is majestic, and it is casual. I find the pronunciation of the back-to-back ‘c’ sounds to be a little difficult.

Main Graphics

crest

stamp

banner

search

Video Style

One of the things we notice is that there are really two different logos. There’s the main one, found on the channel icon and stamped on each video, and the stylized formal one in the banner. I’ve chosen to call this duology, which is common, the House Crest / Stamp duology. The House Crest contains the short form of the channel name, and may also contain some other designs. The Stamp tends to contain the full name.

To throw things off immediately however, Majestic Casual reverses the use of their crest and stamps - they use their crest for everything, and they only use their stamp for the channel banner.

Majestic Casual does not include any song or artist information in their thumbnails, they only show their own name.

Majestic Casual also follows a relatively common pattern among some music channels of making their crest just simple text with simple font. The rest of this list won’t include any more examples of that, but it is common, presumably among people running channels who do not have any design chops.

Video 1

https://www.youtube.com/watch?v=z1iHqt0-zs0

Majestic Casual doesn’t do animations, and their video graphics are on a what-you-see-is-what-you-get basis with respect to the video thumbnails.

Case in point:

Thumbnail:

Video:

Majestic Casual’s video graphics break down into showcasing either a

A) handsome man
B) hot girl, or
C) shot of nature, featuring either A or B or neither.

We see here demonstrated Type A.

Video 2

https://www.youtube.com/watch?v=glK5DW9070w

Static shot, mostly of nature but there’s a cute girl in the middle too.

Video 3

https://www.youtube.com/watch?v=R3zj_c272YM

Static shot of hot girl.

Style Summary

Majestic Casual is the easiest of all graphic types to mimic. Easy crest, easy stamp, and the only work they put into their video graphics are changing up the background image. For the lazy among is, this is the way to go.

As far video graphics go too, they’re an easy style to consider. Grab some images of scenery or attractive girls from the internet and go to town. Majestic is a channel of minimal effort.

The main problem with them is one of recognizability. At a glance, Majestic looks generic, although it is likely that everyone else copied them, given how early they were to the scene.

Video Graphic Information

Majestic Casual includes their branding in each frame of the graphic.

They do not include song information.

They do not include artist information.

They do not include genre information.

Additional Graphics

Included in the Majestic stamp is this lion emblem, which contains the year they were established.

I’m not sure how important this is to their branding, it kind of fades into the background in every video. If they dropped it completely from their branding I don’t think they’d suffer in any capacity.

About

From their About page:

Majestic Casual was born out of pure appreciation for the things we love.
We don’t care about trends, genres, stats or names. We value authenticity and diversity, things that spark our memories and emotions. To find these rare experiences we stay curious and always with open ears and eyes for the undiscovered.
Majestic Casual is a carefully selected collection of things we admire. Our very favorite discoveries shared with curious minds from around the world – the list of recommendations we’d give to our friends who seek for some good advice beside the commercial and algorithm driven mainstream.

Additional notes

Majestic has gone all out on the other social media site game. An active twitter (last post October 16), an active SoundCloud (last repost October 15), an active Facebook (last post October 13), an active Instagram (last post October 17), an actively maintained Spotify playlist (last updated October 15), and Snapchat.

What these guys don’t have is a real website, although they do have a store, and the main domain redirects to their YouTube channel.

Website

Although they own the majesticcasual.com domain, it’s just a redirect to their youtube channel.

TheSoundYouNeed

Name

A bold but smooth choice of name. They got what you need, and it’s some good sound. Stylistically they chose to run their name together, which makes it easy to search for and gives it that “you know it when you see it” quality.

Main Graphics

crest

stamp

banner

search

Video Style

We see another example of the Crest / Stamp duology.
Their crest is a nicely designed symbol that contains their shortened channel name TSYN, for The Sound You Need, along with when they were created (2012, apparently).

Their crest is also the perfect size to be used as a user-icon, which in recent trends is a circular crop of a rectangular image.

Their stamp, a somewhat loopy looking font writing out their full name without spaces, TheSoundYouNeed also contains their crest beneath it in a smaller font.

TheSoundYouNeed does not include any artist or song information in their thumbnails.

Video 1

https://www.youtube.com/watch?v=f4VmOSgTL6c

This is really involved - the channel stamp is animated out like handwriting, and the background is constantly sifting between different types of pastel colored kaleidoscope fractals.

Video 2

https://www.youtube.com/watch?v=Exg0Vmkf190

Unlike the first one, this is a classic TSYN video graphic - no animations, no fancy fractals, just some brightly abstract piece of art, stamped with their logos.

Video 3

https://www.youtube.com/watch?v=pdwVANZ5LWc

This is like a mix of the previous two styles - there’s no handwriting intro, but there are kaleidoscope fractals. Still sticking to the pastel theme here, too.

Style Summary

In terms of artistry-per-upload, I don’t think anyone on this list will match TSYN. Who among us has the patience to make fractal geometry for each video, that also seems to be beat-aligned with the music? Not me. On the other hand, they’re extremely pleasing to look at, and no doubt is responsible for a lot of captured audience attention over the years.

That handwriting into they sometimes do is also a neat effect, and although more doable than the fractals, because it can be reused, will require some thinking if we choose to go that route.

TSYN generally chooses an uplifting pastel color theme, which is nice.

As far as their logos go, I find their crest to be uninspired, but their stamp has a really nice casual feeling to it.

Video Graphic Information

TheSoundYouNeed includes their branding in each frame of the graphic.

They do not include song information.

They do not include artist information.

They do not include genre information.

About

Subscribe to TSYN and discover The finest music of our time.
✙ Bass ✙ UK House ✙ Future Garage ✙ Lounge music ✙ Hip-Hop ✙ Minimal ✙ & More..
If you want to stay even more up-to-date, you can track us on Facebook, Tumblr, Soundcloud, Twitter and Instagram
- - - - - - - - - - - -
- - - - - - - - - - - -
Contact Us
TheSoundYouNeed
Third Floor, Bankstock Building,
42-44 De Beauvoir Crescent,
London, N1 5SB
Unites Kingdom
(Disclaimer) We do not own all of these songs. All songs found on the YouTube Channel TSYN are for promotional reasons only. Any artistic perspective portrayed in any of the songs comes from the artist of the song only. No views expressed in the songs come from TSYN, Only the artists themselves.

Website

Going by their shortened name, they own the domain tysn.co, although it contains nothing but a splash page.

MrSuicideSheep

Name

I think once upon a time I read that the channel owner just chose the name one day during their teenage years and it became so much more than they could ever have hoped for. This would certainly explain its Invader Zim / Newgrounds-y kind of vibe.

It’s not terrible branding by any measure, though. Through longevity alone it’s become an icon of the Youtube Music field, but also it’s as unique as it gets. If we were choosing our name with the explicit intention of creating a brand from it, it’s likely not the name we would choose, but it’s also not precedent. See also, Death Row Records.

Main Graphics

crest

stamp

banner

search

Video Style

MrSuicideSheep’s House Crest is the grey-tinted sheep, and the Stamp is the black or white Suicide Sheep text. Sometimes this Stamp logo is big, and centered on the thumbnail, other times it’s small and in the corner of the picture.

What’s notable here is that although the channel is MrSuicideSheep, the curator seems to have dropped the Mr from the formal title, as everything is Stamped with just Suicide Sheep.

MrSuicideSheep does not elect to show song or artist information in the thumbnail.

Video 1

https://www.youtube.com/watch?v=pKGn7OiBU_8

Although the primary graphic is of an attractive, vaguely anime looking girl, there’s a modicum of animation with with the blurry black fade in to the channel stamp, following by a blurry fade in to an increasingly crisper still-image.

Video 2

https://www.youtube.com/watch?v=ig3YIvO0nmI

Like can be inferred from the video thumbnail list, MrSuicideSheep’s style is all over the place when it comes to the backing video graphic. This one uses some really nice art of a starry sky.

Video 3

https://www.youtube.com/watch?v=MCVi8Rxig90

Back to girls for the graphic.

This one actually has lyrics at the bottom, but due some video capture errors, I was not able to record the lyrics. But that’s one interesting way to do it, if you happen to feel like putting in extra effort and also know the lyrics to the song.

Style Summary

As far as video graphics go, MrSuicideSheep has a big variety - some are uplifting, cute girls; others are sprawling vistas; others still are drawings and paintings at varying degrees of abstraction. It almost looks like whatever the channel curator happens to be feeling at that particular moment. There is no overarching, cohesive sense of style to the graphics.

I also find the channel crest to be very strange. It is such a light shade of gray, with such a nondescript graphic in the middle, that my first impression is that of a thumbnail placeholder while some other asset loads in. Of course that is not the case and that is the full graphic.

Notably that graphic is almost not used - far more widespread if the stamp, containing the slightly loopy font spelling out their name.

Additionally, although the channel branding is relatively visible in the thumbnails, it is really hard to find in the video graphic beyond the first 5 seconds of the intro animation. If you look carefully you can see it in the bottom right corner, but the focus is clearly on the good pictures instead.

Video Graphic Information

MrSuicideSheep includes their branding in each frame of the graphic, but in a very small and unobtrusive way.

They do not include song information.

They do not include artist information.

They do not include genre information.

About

On this channel you will find a wide variety of different electronic and sometimes non-electronic music. I strive to find the best and most enjoyable music for you guys. I hope you have a good time here :)
Please submit your music here.
http://submit.mrsuicidesheep.com/

Website

https://www.mrsuicidesheep.com/ is an owned domain, but it just serves as a portal to advertise branded shirts.

Chill Nation

Name

Like the other *Nation channels, this channel’s name evokes images of togetherness, comradery by being members of the nation. It reminds me of Red Sox Nation, which is the same thing but for fans of that baseball team.

The Chill in this particular channel represents the kind of music they post. They’ll post songs of significantly different genre to one of their many other channels.

Their final branding score? 7/10

Main Graphics

crest

stamp

banner

search

Video Style

Although Chill Nation follows the Crest / Stamp duology, the Crest half isn’t much different from the Stamp half.

Both spell out Chill Nation in the same font, the only difference is the font and background color. Additionally, the Stamp half has some palm-trees in the bottom, which is a nice touch of style.

I actually wonder why they don’t just standardize on the Stamp for all their logos, it’s much better than the Crest.

Chill Nation does not elect to show song or artist information in the thumbnail.

Video 1

https://www.youtube.com/watch?v=03GwlsRZnMY

Chill Nation tends to go for either some kind of vaguely interesting but structureless colored background, or for some epic vista of nature as their background.

They have chosen the vista.

This is also our first introduction to the sheer production of the Trap Nation network – Smooth crest fly in from off screen displaying all their other channels for a second, the subtle, constant swaying motion of the palm trees inside that crest, the slight pulsing of the snowflakes to the beat of song, and once the song gets underway, in addition to all of the above, they map the waveform to a kind of oil-colored pulse on the circumference of the crest.

trap Nations love for animation is abundantly apparent, and it’s generally a pleasure to look at.

Video 2

https://www.youtube.com/watch?v=iAakgRlbJok

Like described above, this is one of their formless color choices of background.

We can see a good example of what the song beats look like when we’re deep into the music. Strong pulses and big animations everywhere.

Plus we can see their smooth “Follow on Instagram” plug that rolls in at the bottom left for a bit before rolling away.

Video 3

https://www.youtube.com/watch?v=OUBrVRcnQds

I neglected to grab a video capture for this track because by this point it’s clear that Chill Nation has no variation. This one was a nature background exactly like the first, and had no other special qualities.

Style Overview

The Trap Nation channels go big on animation, and they put in effort for each video to make it pleasant to look at with the waveform pulsing. Their background choice leaves something to be desired though, because aside form the nature vibes, each video has a really “samey” kind of feel to it. All that is different is the strength of the waveform.

These guys are great at branding though. Stamp in the bottom corner, Instagram banners poking out occasionally, the quick “did you know we had all these other channels” showcase at the beginning of each video. It’s all quite fantastic.

Video Graphic Information

Chill Nations branding is front and center, the literal focal point of each piece, plus their sister channel showcasing at the start, and just in case you forgot, their stamp is the bottom righthand corner as well.

They do not include song information.

They do not include artist information.

They do not include genre information.

About

Welcome to Chill Nation! This channel is the official second channel of "Trap Nation", at Chill Nation we upload more chill and laid back music you won't hear on Trap Nation.

Website

Chill Nation delegates their website to their parent Nations, at https://nations.io/pages/chill-nation.

These guys actually have legit websites with descriptions, proper links, and a dedicated shop.

Eton Messy

Name

What’s in a name? Not necessarily anything in particular, apparently. Although it sounds like someone’s name, it is no such thing. They’ve gone for uniqueness above all else - certainly no google search will ever be confused as to what you mean when you ask it for Eton Messy, or any potentially mistyped variation thereof.

It leaves them with few obvious choices to create any kind of visual link with their name, but there’s a certain freedom that comes with that as well, in that they can make anything at all their brand.

Main Graphics

crest

banner

search

Video Style

Speaking of standardizing on one logo, Eton Messy definitely standardized on the Crest, and it looks good. Their video stamps prominently include their formal name, but the real draw is the Crest.

Like every other channel, Eton Messy does not show the song or artist name in the thumbnail.

Video 1

https://www.youtube.com/watch?v=e6hQand8MjM

A pleasant looking natural background, with a smooth ripple like animation for the fade-in of the stamp, which contains no text except for the initials of the channel.

Video 2

https://www.youtube.com/watch?v=qs0biWHy5WU

An abstract piece, pleasant to look at, with the stamp ripple fade-in.

Video 3

https://www.youtube.com/watch?v=O9N-0krx57k

A pleasant looking tropical background with the stamp ripple fade in.

Style Overview

Although the animation of the logo is nice, it becomes clear that Eton Messy puts less effort into the video once the fade-in has been completed. This is fine, this puts them closer to Majestic Casual than, say, MonsterCat, and that’s perfectly okay.

They tend to go for eye-catching backgrounds, the type you’d elect to make your desktop wallpaper.

If their main logo weren’t enough to tell you who you were watching, a faded version of it can be found in the bottom righthand corner.

Graphic Overview

Eton Messy does not include any information in their video graphics, only the channel stamp.

About

Eton Messy is a Youtube channel turned Record label, which has included dance music events and an international DJ outfit, playing various genres within dance music, including disco, house, deep house, tech house, progressive and funky etc.
For DJ bookings/Event enquiries contact:
tim.levy@mn2s.com
Latest playlists ⟼ Spotify http://bit.ly/EMspotify
Latest news ⟼ Facebook http://on.fb.me/R5QGH6
Latest mixes/music ⟼ Soundcloud http://bit.ly/LMFAUi
Latest chat ⟼ Twitter http://bit.ly/PAcpsh
Latest images ⟼ Insta http://bit.ly/2vC0CRS
Track Submissions ⟼ etonmessysubmissions@gmail.com
DJ Promo list ⟼ https://goo.gl/forms/ygDJf9XUoKSLjJXw2
Business (no music please) ⟼ info@etonmessy.com
For DJ bookings/Event enquiries contact:
info@etonmessy.com
For any copyright issues please get in touch: info@etonmessy.com
Enjoy!

Website

Eton Messy are the first guys on this list to have a truly functional website, although it’s a bit busy and overly complex.

They have their latest YouTube posts in a step-through queue at the top, a list of upcoming events they’re sponsoring or are otherwise interested in, featured news, and a decent about page.

Eton Messy gets points for having a real website, but they really need to tone down the busyness.

Monstercat

Name

These guys know how to have fun. Their brand is that of a goofy, cartoony, cat-thing. Some kind of monster cat.

No problems with uniqueness here. Even in obscurity, just adding “music” would have gotten the proper results, and in their immense popularity they completely dominate anything even related to the term.

Plus it opens the avenue for some fun graphics that link into the name.

Main Graphics

crest

banner

search

Video Style

And now for something completely different.

Monster Cat has a funky, distinct and silly logo. High contrast, designed by someone with some humor. You’ll never mistake this logo for any other logo, which one might reasonably do with nearly everything else on this list.

They don’t even have a regular stamp - sometimes they’ll include the album art of the album of the release they’re showcasing, but they take a real different approach to how they stamp their video thumbnails:

Song info is front and center, below a neat waveform visualization of the track in question. Below that in the left corner is a small crest icon, just enough that you’ll know what it is when you see it, but not big enough to make the channel the main focus. For Monstercat, the music is focus, not the Monstercat.

Style Overview

Because Monster Cat is now a major record producer, they have access to a lot of awesome album art assets, so they can do some awesome animation with the album art of the artist they are showcasing.

The background of each MonsterCat video is the album art, animated in some capacity, either lightly, like in Video 2 and 3, or heavily, like in Video 1.

The most prominent graphic is the standard audio volume meter, meaning you can pretty easily track and follow along with the song just visually.

The thing that sticks out the most is how respectful MonsterCat is of the artists:

the Artists name is the largest piece of text on the screen
The song name is always displayed
In addition to the album art as background, the album art gets to replace the primary MonsterCat logo
And the artist name gets to replace the MonsertCat logo too, it flips around a lot

This is super nice, and respectful to both listeners and artists

Graphic Overview

MonsterCat chooses to prioritize artist and song marketing over themselves

They do not display genre information in the videos, but their video descriptions all contain #hashtags of the genre, such as #dubstep, #heavyedm, etc.

About

Unbound, Unlimited, Uncaged! Monstercat breaks the barriers of expectations with a movement that will make 2018 the biggest year in our history.
Monstercat is a new-age Record Label. It is the convergence of an independent label, tech startup and media company. We empower creative people, musicians, artists, developers, entrepreneurs, and influencers young and old. We looked at the problems within the music industry, asked “why not” and have created the “how to”.
If you would like to use Monstercat's music in your own videos or Twitch streams:
https://license.monstercat.com

Website

https://www.monstercat.com/

Although their branding is goofy, Monstercat doesn’t joke around. Their website is the real deal, professional to the core. Their music is front and center, but their artists, publishing information, blog and other aspects are all available at the click of a button.

MikuMusicNetwork

Name

If their video thumbnails didn’t give them away their name sure does. These guys are weebs. Their music taste is alright though.

They don’t really live up to their name of being a music network, that’s much more like Chill Nations thing, but they get close enough.

Main Graphics

crest

stamp

banner

search

Video Style

Speaking of completely different, once again here’s something different. These guys are taking a new spin to the Crest/Stamp duology. There’s a base, which is fractal diamonds, and then they switch up what’s inside. The crest has a Miku silhouette with a stylized ミク in the middle.

As for the stamps, there are actually two, The first takes the base crest and drops the middle image, leaving room for cute anime girls to pop out and be framed by the fractals.

The second stamp is a more traditional take with the name, the short form in this case, written out along with a splashy icon.

It seems to be random whether one stamp is used over the other.

The thumbnails take another interesting twist - Instead of prioritizing either the channel name or the artist, the thumbnails prioritize the genre. Each thumbnail is, to varying degrees of clarity, tagged with Electro, Drum & Bass, Future Bass, Lovestep. That’s an interesting take on the thumbnail concept, and I quite like it.

Additionally, even without the channel name, it’s clear where the video is coming from. The concept, style, and execution of the genre tags are so unique, as well as the choice of vivid, bright color pallet anime girls makes it obvious.

Video 1

https://www.youtube.com/watch?v=oFbljhlRmIQ graphic: cute anime girl / with icon

These guys seem to have an animation budget, although they also seem to have some strange tastes. Each of their videos is a bit jumpy because there’s this constant rocking motion of the frame.

Putting the rocking aside, the graphic is actually super cool. The rotating squares around the center, and the two diamonds showing the name of the artist, and the song are really neat.

And then their choice of cute anime girl, in this case a catgirl, gets a cool stripe fade in.

Their stamp is on the bottom, after the crest gets replaced with the catgirl.

Video 2

https://www.youtube.com/watch?v=53vGcMbQAuQ

This is the second of two main styles that MMN opts for. This format opts for an entirely different set of logos for themselves, though they kept their animation budget.

Their Crest becomes the stylized Miku(ミク), and their stamp becomes this themed Hatsune Miku paint splash.

They stick to their cute anime girl aesthetic, but don’t opt for displaying any information about the song.

As usual, the rocking continues.

Video 3

https://www.youtube.com/watch?v=Ahlji3K5MuQ

The third video follows the first - rocking, a cool anime girl fade-in, and rotating diamonds.

Style Overview

When in comes to the first of their video styles, I find the MikuMusicNetwork to be just as respectful to the artist as MonsterCat is: Their priority is to showcase cute anime girls, but the artist and the song title are on prominent display.

However, style 2 is focused on the network, and doesn’t showcase anything about the song in question.

Their animation budget is evident, and appreciated, but they really need to do away with the constant rocking. It’s just weird and distracting.

Graphic Overview

MMN sometimes chooses to display artist information, and sometimes doesn’t.

Notably, the thumbnails actually contain the information all the time, with additional genre information, as available,

It would be nice if this information was there all the time.

Their logo is always present, either in the intro fade-in, or at the bottom using one of their two splash icons.

About

Welcome to MikuMusicNetwork, a Anime inspired music promotion channel that focuses on bringing smaller artists and bigger ones alike to one convenient place for you to listen to, from the hardest of drops to the chillest of vocals!
Want your track to be uploaded or do you want to label your latest release with us? That is possible! Click on Track Submissions and fill in the form. We'll respond as soon as we can!
Be aware that we get plenty of submissions and we are unable to respond to all of them! We are working currently on a better method that allows us to respond to all submissions!
Music Producer or Label? Have an issue regarding a video hosted on this channel? Please contact us through PM and we will respond as quick as possible!

Website

There is no website associated with this channel.

Music Channel Part 2 - Stats

Tue, 26 Mar 2019 15:50:58 -0700

This is the second in a series of posts about analyzing YouTube music channels.

Part 1 - Music Channel Introduction and Data Collection
Part 2 - Analysis of Channel Activity
Part 3 - Branding

Your Questions, Answered

We left off last time with a freshly created database full of information we could query. We sought to answer a few questions:

How many videos should we seed our channel with?
How often should we upload?
Should we upload more than video at once?
How long can we wait between uploads if something goes wrong?

Channel Seeding?

We’ll take the first upload date of each of our channels and see what their total upload count is:

SELECT Channel, Count FROM ChannelPublish WHERE DaysSinceLastPublish=0 ORDER BY Count DESC

Which returns the following data:

Channel	Count
Majestic Casual	6
ChillNation	1
Eton Messy	1
MikuMusicNetwork	1
Monstercat: Uncaged	1
MrSuicideSheep	1
TheSoundYouNeed	1

With the exception of Majestic Casual, no one seems to seed their channel. That makes things easy on us.

Question: How many videos should we seed our channel with?
Answer: None

Upload Pace?

SELECT
 cp.Channel,
 ROUND(avg(cp.DaysSinceLastPublish),2) AS AvgDaysBetweenPosts
 FROM ChannelPublish AS cp
 WHERE DaysSinceLastPublish > 0
 GROUP BY cp.Channel

Channel	Average Days Between Upload
ChillNation	1.72
Eton Messy	3.50
Majestic Casual	1.52
MikuMusicNetwork	2.81
Monstercat: Uncaged	2.04
MrSuicideSheep	1.34
TheSoundYouNeed	6.65

This might not tell the whole story though, so lets see if we can get some sense of upload pace vs channel popularity.

We’ll look at a channels lifetime views, along with the average number of views per song per channel. If we take all this information together, we’ll get a stronger sense of where upload pace fits into the scene.

SELECT
 cp.Channel,
 ROUND(avg(cp.DaysSinceLastPublish),2) AS AvgDaysBetweenPosts,
 (
 SELECT SUM(vs.Views) FROM VideoStats AS vs WHERE vs.Channel = cp.Channel
 ) AS TotalChannelViews,
 (
 SELECT CAST(AVG(vs.Views) AS INTEGER) FROM VideoStats AS vs WHERE vs.Channel = cp.Channel
 ) AS AvgViewsPerUpload

 FROM ChannelPublish AS cp
 WHERE DaysSinceLastPublish > 0
 GROUP BY cp.Channel
 ORDER BY AvgViewsPerUpload DESC

Channel	Average Days Between Upload	Total Channel Views	Average Views per Upload
TheSoundYouNeed	6.65	1,965,150,738	6,102,952
MrSuicideSheep	1.34	3,912,749,521	1,956,374
ChillNation	1.72	1,731,630,505	1,777,854
Monstercat: Uncaged	2.04	2,164,378,038	1,592,625
Majestic Casual	1.52	1,704,478,658	893,801
Eton Messy	3.5	113,665,075	143,155
MikuMusicNetwork	2.81	12,827,204	12,453

These stats are a bit wonky because each channel tends to have 3 or 4 super-outliers. For example, TheSoundYouNeed’s Tom Odell - Another Love (Zwette Edit) has 395 million views, which is a full 20% of that channels views. There’s more statistical magic we could do involving outlier removal and standard deviations, but we’re just going to leave it is and say that we should try to target one upload every one or two days.

Question How often should we upload?
Answer: Every day, or every other day.

Batch Uploads?

SELECT Channel, Count AS BatchSize, COUNT(Count) AS TimesUploaded FROM ChannelPublish GROUP BY Channel, Count ORDER BY Channel, TimesUploaded DESC

Channel	Batch Size	Times Uploaded
ChillNation	1	914
ChillNation	2	30
Eton Messy	1	722
Eton Messy	2	30
Eton Messy	3	4
Majestic Casual	1	1318
Majestic Casual	2	159
Majestic Casual	3	38
Majestic Casual	4	18
Majestic Casual	5	7
Majestic Casual	6	3
Majestic Casual	7	2
Majestic Casual	9	2
MikuMusicNetwork	1	474
MikuMusicNetwork	2	269
MikuMusicNetwork	3	6
Monstercat: Uncaged	1	1247
Monstercat: Uncaged	2	56
MrSuicideSheep	1	1938
MrSuicideSheep	2	31
TheSoundYouNeed	1	302
TheSoundYouNeed	2	7
TheSoundYouNeed	3	2

As we can see, on days that a channel uploads, they only upload one song. There’s a small amount of variation within most channels, and the MikuMusicNetwork is an outlier that uploads twice a day roughly a quarter of the time, but it seems with only some exceptions that once-per-upload-day is the norm.

Question: Should we upload more than video at once?
Answer: No.

Upload Gaps?


SELECT
 cp.Channel,
 Max(cp.DaysSinceLastPublish) AS MaxDaysBetweenPosts,
 ROUND(avg(cp.DaysSinceLastPublish),2) AS AvgDaysBetweenPosts,
 (
 SELECT SUM(vs.Views) FROM VideoStats AS vs WHERE vs.Channel = cp.Channel
 ) AS TotalChannelViews,
 (
 SELECT CAST(AVG(vs.Views) AS INTEGER) FROM VideoStats AS vs WHERE vs.Channel = cp.Channel
 ) AS AvgViewsPerUpload

 FROM ChannelPublish AS cp
 WHERE DaysSinceLastPublish > 0
 GROUP BY cp.Channel
 ORDER BY AvgViewsPerUpload DESC

Channel	Max Upload Gap	Total Channel Views
ChillNation	149	1731630505
Eton Messy	178	113665075
Majestic Casual	13	1704478658
MikuMusicNetwork & Records	59	12827204
Monstercat: Uncaged	14	2164378038
MrSuicideSheep	36	3912749521
TheSoundYouNeed	56	1965150738

We’ll quickly drop into SQLite DB Browser’s ploting mode to graph Upload Gap (x) vs Views (y):

We see a correlation between fewer days between uploads and higher total view counts, but again this is basically raw data that has quirks like super-videos going uncompensated for.

Of course this basically fits with our impression of YouTube, right? Regular uploads are better than sporadic uploads.

Question: How long can we wait between uploads if something goes wrong?
Answer: Try to keep it below 150 days, but don’t sweat it.

Summary

So in summary we’re not going to seed our channel, and we’ll aim to upload about 3 or 4 times a week to start off with.

Now that we have a sense of what we need to be committing to regarding our upload pattern, we can move on to the more fun aspects of channel curation, such as picking out a name and a logo.

And once that is done, we’ll start picking out our initial upload set.

Music Channel Part 1 - Data

Mon, 25 Mar 2019 18:17:32 -0700

This is the first in a series of posts about analyzing YouTube music channels.

Part 1 - Music Channel Introduction and Data Collection
Part 2 - Analysis of Channel Activity
Part 3 - Branding

Background

I often listen to new music on YouTube because their discovery systems unearth some occasionally great finds. After a while of listening there however you begin to realize that there are only a handful of channels that get recommended to you, and they all seem to just upload songs and let the views roll in, not unlike a radio DJ. Sometimes these channels become so successful that artists will begin to sign with the channels for exclusive releases, in effect turning the channels into a de-facto label. Some of these channels have even turned into de-jure labels, such as Proximity.

I thought it would be a fun experiment to see if I could create a YouTube channel of my own doing the same thing these channels were doing. It seemed easy enough, just upload a few songs here and there. But before I began, I wanted to get a sense of what, exactly, these channels were doing, what their upload schedule was actually like, what were they like when they started out, how did they evolve, etc.

This series of posts describes just such an analysis of a selection of music channels on YouTube.

Data Collection

First, I wanted to get a sense of how these channels upload songs, which would help inform me on how run my own channel. The questions that I wanted answers to before starting were:

How many videos should we seed our channel with?
How often should we upload?
Should we upload more than video at once?
How long can we wait between uploads if something goes wrong?

YT Tracker

To answer these questions, I had to get some data first. I wasn’t about to scrape YouTube myself, so after some googling for some tools that could help me gather some channel stats, I came across the excellent YT Tracker, an addon for Google Sheets that pulls all kinds of channel information for you.

You can find its full functionality detailed here.

Channel List

First I seeded YT Tracker with a list of 7 of the channels I either use or happen to come across often:

As the developer of YT Tracker writes on his page, it can sometimes be tricky to extract the channel name from some pages. If you’re on a page for a given video, the main uploader link in the video description will contain the channel name - this goes for accounts that are flagged as “user”, as well for for “channel” accounts. This is important because YT Tracker only operates on channel ids.

Adding to a YT Tracker Sheet

We add the channel ids to the first column:

YT Tracker takes care of filling in the rest of the data automatically. It’s really quite magical.

Grabbing Video Data

Next we want to grab video data from each channel. With the channel id cell of the channel we want to grab highlighted, we use the Add-Ons drop down menu to select YT Tracker and select Fetch Channel Videos, which will create a new sheet for us, populated with video upload data:

You have to do this highlight-menu-fetch process for each item in our list.

Exporting

Next we’ll take each sheet and Save-As .csv to some folder on our computer.

This is the first step to getting the data into a SQL-queryable format.

Data Cleanup Part 1

Every project that involves data will require some form of cleanup process, this is no different.

CSV Cleanup

Adding Channel Id

Because we’re going to end up putting all this data into SQLite, we need to do some work on the CSV files first. To make our data amenable, one of the things we’ll do is add the channel name to each row of each CSV.

For example, we’re going to turn:

Video Id,Title,Thumbnail,Publish On,Duration,# of Views,# of Likes,# of Dislikes,# of Comments
x8Y2iCSGSXI,Nora Van Elken - Best I Ever Had,,2018-10-11,3 : 02,"58,593","3,082",78,141

into

Video Id,Title,Thumbnail,Publish On,Duration,# of Views,# of Likes,# of Dislikes,# of Comments
ChillNation,x8Y2iCSGSXI,Nora Van Elken - Best I Ever Had,,2018-10-11,3:02,"58,593","3,082",78,141

I did this with the following two regexs:

find: ^([0-9A-Za-z_-]*)?,
replace: ChillNation,$1,

where ChillNation is whatever channel you’re actually working with.

Be sure to add the Column name of ChannelName before Video Id.

Patching up Time

The timestamps are bit messy - consistent, but full of weird spaces that make parsing a bit of a pain.

We want to turn

3 : 02
1 : 26 : 27

into

3:02
1:26:27

We fix this with the following regex find/replaces:

find: ([0-9]{0,2})\s+:\s+([0-9]{0,2})
replace: $1:$2

Which we run twice to catch any hour+ long tracks with hh:mm:ss formatting.

Changing Column Names

The exported named columns would make for some difficult SQL querying. To make our life easier, we’ll change the names:

Channel,VID,Title,Thumbnail,Published,Duration,Views,Likes,Dislikes,Comments

SQL Part 1

I like to do as much data processing and analysis in python and sqlite as possible. We’ll do that here too, by importing all our exported files into a new Sqlite database.

I prefer to use DB Browser for SQLite, which is a Windows GUI tool.

Using the Import -> Table From CSV File menu option, we add in all our exported csv files, saying okay to the append to table? warning. You can call the new table whatever you like, I’ve chosen VideoStats.

Table

After this important, we end up with a table that looks something like this:

While a good first step, I realized here that the data wasn’t actually in the most useful form. Views, Likes, Dislikes, and Comments all have commas in their data, which makes them strings, not integers.

Additionally, Duration has colons in it, which makes it a string too.

The exception is the Published field, which SQLite handles correctly as a date.

Adding a Duration_Ms Column

Although I want to fix the duration field, I think it might be useful to keep its original form around because its very human-readable. So we’ll add another column to our table, Duration_Ms, which will represent the duration of the song in milliseconds:

Python Cleanup Part 1

We’ll use Python here to remedy these errors. This is python at its best - small scripts for data processing.

import os,sys,sqlite3,pathlib
from dateutil.parser import parse as PDT
from datetime import datetime, timedelta

sqlfile = "./ytrackerstats/ytstats.db"

def fixDuration(dur):
 try:
 s = dur.split(':')
 s.reverse()
 TotalSeconds = 0
 for idx,num in enumerate(s):
 if num == '':
 num = 0
 multiplier = 60 ** idx
 number = int(num)
 seconds = number * multiplier
 TotalSeconds += seconds
 return TotalSeconds * 1000
 except ValueError:
 print(dur)
 raise Exception("???")

def fixCommaNumber(com):
 try:
 if type(com) is str:
 if com == "undefined":
 repl = "0"
 else:
 repl = com.replace(',', '')
 i = int(repl)
 elif type(com) is int or type(com) is float:
 i = com
 return i
 except ValueError as e:
 print(com)
 print(e)
 raise Exception("??")

def FetchAllFromVideoStats():
 connection = sqlite3.connect(sqlfile)
 c = connection.cursor()
 c.execute("SELECT VID,Duration,Views,Likes,Dislikes,Comments FROM VideoStats")
 newRs = []
 for row in c:
 vid = row[0]
 dur = row[1]
 view = row[2]
 like = row[3]
 disl = row[4]
 comm = row[5]
 fixed = (fixDuration(dur), fixCommaNumber(view), fixCommaNumber(like), fixCommaNumber(disl), fixCommaNumber(comm), vid)
 newRs.append(fixed)

 return newRs

def FixVids(tups):
 connection = sqlite3.connect(sqlfile)
 c = connection.cursor()
 q = "UPDATE VideoStats SET Duration_Ms=?, Views=?, Likes=?, Dislikes=?, Comments=? WHERE VID=?"
 c.executemany(q, tups)
 connection.commit()
 return

def test():
 duration = "03:01:46"
 assert fixDuration(duration) == 10906000
 comma1 = "5,600"
 assert fixCommaNumber(comma1) == 5600
 comma2 = "56"
 assert fixCommaNumber(comma2) == 56
 comma3 = 100
 assert fixCommaNumber(comma3) == 100


def main():
 test()
 vids = FetchAllFromVideoStats()
 FixVids(vids)

if __name__ == "__main__":
 main()

The summary of this script is that we remove commas from any numbers, and convert durations to their ms representation, which we update into the Duration_Ms field.

SQL Part 2

Now that most of the data was in SQL, and in a good format, I started running some queries. For instance:

SELECT Channel, Published, COUNT(Published) AS Uploaded FROM VideoStats GROUP BY Channel,Published ORDER BY Channel, Count(Published) DESC

Which returns the number of tracks uploaded per channel per day. This is a useful query - Do people typically upload more than one track per day? Who knows? Well now, you do! With the power of Group By.

Another Table

The problem is that some of these queries started to get a bit cumbersome and unwieldly. Running multiple versions got tiring pretty quick. The soluton was another table. I called this one ChannelPublish.

I also wanted another piece of data, which would have otherwise been really difficult to extract using just plain SQL: Days since last publish.

It seemed important to know how many days had passed between consecutive uploads. This is information that can be derived from the data as it exists, but it’d be a lot nicer if we had a dedicated field for it. This process is known as Feature Engineering, and is common when working with tabular data.

Select Insert

To get data into our new table, we run the following query:

INSERT INTO ChannelPublish2 (Channel, Published, Count)
 SELECT Channel, Published, COUNT(Published) AS Uploaded
 FROM VideoStats
 GROUP BY Channel,Published
 ORDER BY Channel, Count(Published) DESC

Which gets us a table composed entirely of the previous group-by query results.

Python Cleanup Part 2

Moving back to python, we’ll update our script to calculate the number of days since the last upload:


#! /usr/bin/python

import os,sys,sqlite3,pathlib
from dateutil.parser import parse as PDT
from datetime import datetime, timedelta

sqlfile = "./ytrackerstats/ytstats.db"


[... Edited for brevity ...]


def DateDiffRaw(prev, date):
 if prev == 0:
 prev = date
 diff = date - prev
 return diff.days

def FetchAddDaysSinceLastPublish():
 connection = sqlite3.connect(sqlfile)
 c = connection.cursor()
 q = "SELECT Id, Channel, Published FROM ChannelPublish ORDER BY Channel, Published ASC"
 prev = 0
 currentChannel = ""
 updates = []
 for row in c.execute(q):
 id = row[0]
 channel = row[1]
 if currentChannel != channel:
 currentChannel = channel
 # RESET!
 prev = 0
 date_raw = row[2]
 parsed = PDT(date_raw)
 diff = DateDiffRaw(prev, parsed)
 prev = parsed
 updates.append((diff, id))
 return updates

def UpdateDaysSinceLastPublish(tups):
 connection = sqlite3.connect(sqlfile)
 c = connection.cursor()
 q = "UPDATE ChannelPublish SET DaysSinceLastPublish=? WHERE Id=?"
 c.executemany(q, tups)
 connection.commit()



def test():
 duration = "03:01:46"
 assert fixDuration(duration) == 10906000
 comma1 = "5,600"
 assert fixCommaNumber(comma1) == 5600
 comma2 = "56"
 assert fixCommaNumber(comma2) == 56
 comma3 = 100
 assert fixCommaNumber(comma3) == 100

 dateEarlier = "2018-10-09"
 dateLater = "2018-10-10"
 d1p = PDT(dateEarlier)
 d2p = PDT(dateLater)
 assert DateDiffRaw(d1p, d2p) == 1

def main():
 # test()
 # vids = FetchAllFromVideoStats()
 # FixVids(vids)
 dates = FetchAddDaysSinceLastPublish()
 UpdateDaysSinceLastPublish(dates)

if __name__ == "__main__":
 main()

SQL Again

Now we have a table of very useful information:

Which lets us easily query for stuff like this:

SELECT
 Max(cp.DaysSinceLastPublish) AS MaxDaysBetweenPosts,
 CAST(avg(cp.DaysSinceLastPublish) AS FLOAT) AS AvgDaysBetweenPosts,
 cp.Channel FROM ChannelPublish AS cp
 WHERE DaysSinceLastPublish > 0
 --AND Published > date("2018-01-01")
 GROUP BY cp.Channel

Summary

This first post covers the initial setup for being able to graph, chart and otherwise ask statistical questions about some YouTube channels that we’re going to use as a basis for our own music channel.

We gathered our initial dataset, cleaned that data, and put it all into a SQLite file for use later.

Next time, we’ll cover the channel setup, and answer the questions like:

How many videos should we seed our channel with?
How often should we upload?
Should we upload more than video at once?
How long can we wait between uploads if something goes wrong?

Once that information is covered, we’ll create a list of songs to upload and check to see if our favorite channels have covered them already. The less channels a song appears on of course, the better for our channel. We’re not just trying to be a mirror of the others, after all.

We’ll also see what we can do about creating a Channel Icon, selecting some backgrounds, adding that must-have waveform pulse animation, and some initial channel curation.