Rebuilding PubHub: A Cautionary Tale of AI-Driven Requirements, Part 4

PubHub is Cisco DevNet’s internal content management system, and this is the fourth part in the series of how the DevNet team rebuilt it from the ground up. In Part 1, we covered the wins and hurdles of rebuilding PubHub, while Part 2 explored how AI accelerated our software development lifecycle (SDLC). Part 3 is about the DevNet UX team’s journey to turn Cisco design standards into AI skills and an MCP server. Now, we are diving into a workflow evolution where we used AI to capture project requirements, acceptance criteria, and more. This shift may have been more than what we bargained for.

The Exhilaration of AI

Because we were rebuilding from scratch, we saw an opportunity to rewrite the new system based on a decade of personal experience. We used AI to help document the requirements for this project using VS Code Copilot paired with Claude Sonnet and Opus.

In the world of Agile development and the SDLC, we use epics to organize complex projects. An epic is a high-level requirement or a large-scale feature that is too big to be completed in a single sprint. It acts as a container for multiple related user stories, providing the necessary context and scope for the engineering team. To keep pace with AI’s speed, we didn’t use traditional document programs like Word. Instead, an AI skill generated the epics and user stories in Markdown and hosted them in an internal GitHub repo. Versioning each change and maintaining a clear changelog as the project evolved allowed us to treat our requirements like code.

The initial results were exciting. AI transformed the story writing process from a chore into a superpower. I could feed AI specific criteria and pain points, and it would instantly generate:

  • Comprehensive Context: Clear business value and scope to prevent scope creep.
  • Technical Precision: Detailed acceptance criteria, actor perspectives, and key concepts.
  • Rapid Prototyping: API request/response models, wireframes, and HTML-based mockups to align engineering, design, and leadership.
AI-generated sections of a user story
AI-generated sections of a user story
Mockup of the Git Access Unavailable notification
AI-generated mockup of Git Access Unavailable notification
AI-generated page of wireframes
AI-generated page of wireframes

AI didn’t just write; it managed. It versioned documents and maintained a meticulous changelog across 12 dense epics, each linked to child stories and their dependencies.

I felt like I had found a cheat code for technical leadership.

The Tipping Point: When “Complete” Becomes “Too Much”

As the documentation grew, I hit a wall. The epics became so dense that even I, the person who prompted them into existence, struggled to review and edit them. For example, I would frequently meet with the engineering team to discuss issues and compromise on launch trade-offs. After the meetings I used my agent in VS Code Chat to update the requirements within the epics. While the tool flawlessly synced updates across our documentation pipeline, it handed me a massive review queue. I refused to blindly approve hundreds of changes, preferring to keep the human in the loop (me) rather than building another agent to handle the review.

This created a paradox where AI gave us perfect consistency, but I paid for it with hours of manual verification.

Chat sessions listed with the number of changes for each session
Number of changes for each chat session

I remember thinking, If AI hadn’t pointed out these tiny dependencies, it would have taken me days to find them manually or miss them completely. But then the realization hit me: I wouldn’t have written this much content in the first place. AI had generated a mountain of content, and I was spending all my time digging through it.

The “Review Tax” became apparent when I shared the meticulously written epics with my colleagues. After sharing the epics with the principal engineer, I heard that the documentation was too prescriptive for engineering. It was too detailed, did not leave enough room for engineering to adapt based on the new system they were building, and most importantly, it would take too long for the whole team to read.

I initially smirked, thinking, Why would they read it when AI can read it for them?

It raised a practical question: Was I creating a loop where I used AI to generate documentation simply so it could be read and implemented by AI? While the content was clear and comprehensive, the sheer volume had become a barrier to the very collaboration it was meant to support.

The Pivot: Writing for Humans

I realized that in my quest for technical perfection, I had forgotten the most important stakeholder: the human.

We pivoted to a lean documentation approach, but we didn’t throw away the epics. Those epics remained the “deep-dive” source of truth, perfect for AI ingestion and technical reference. Instead, we created summaries. These were bite-sized, high-level references that allowed engineers, designers, and managers to grasp the big picture, understand key workflows, and identify related and dependent epics without drowning in detail. By creating parallel “AI-readable” epics with “human-readable” epic summaries, we restored the balance between technical depth and organizational agility.

AI can exponentially increase the amount of output we produce, but it cannot replace the human need for clarity and brevity. If we aren’t careful, we will spend more time managing the output of our tools than we do leading our teams.

Going forward, I recommend a lean approach to producing documentation, whether it be epics, user stories, or strategy docs. When we find ourselves in awe of what AI can generate, we must remember to pause, look up, and ask ourselves: In the rush for productivity, who is leading whom?

Join the Next DevNet Event

Join the Next DevNet Event

See the list of events

Leave a Comment

x
1
1
Voices are browser-dependent.
Tip: Chrome provides the most options.