Blog #48: Slow AI UX Fail – When User Experience can't wait for Artificial Intelligence

Team size: 5 people. Deadline: 2 weeks to integrate an LLM (Large Language Model) into the customer support system. The project was at the startup stage, requiring speed and flair to attract users. We were excited to launch the AI Q&A feature, but right on the first day, the bounce rate spiked inexplicably.

The Problem: The "Black Hole" of Silence

The issue was: A complex AI response could take 10-15 seconds to process on the server side. During those 15 seconds, my User Interface (UI) just showed a soulless spinning Spinner.

Users in the 4.0 era do not have enough patience to wait 15 seconds without seeing any signs of progress. They assumed our application was frozen and hit the exit button. We had a very smart "Artificial Intelligence," but it was obscured by an outdated "User Experience."

Options Considered

We stood between 2 handling directions:

Option 1: Using Long Polling & Partial UI (The Patch)

• Solution: Break down the user's question into parts, or display simulated notifications: "AI is thinking...", "AI is looking up data...".
• Pros: Easy to set up based on the existing old HTTP REST structure.
• Cons: Still doesn't solve the root of the problem. Users still perceive the wait as "fake."

Option 2: Transitioning to Server-Sent Events (SSE) & Streaming (The Real Solution)

• Solution: As soon as the server generates the first word (token), it sends it immediately to the Frontend. Words will appear gradually on the screen as if someone is actually typing.
• Pros: Users see results almost immediately (within 500ms). The application feels "alive" and smarter.
• Cons: Must scrap the old API logic. Harder UI handling (autoscroll, markdown parsing while streaming).

Final Decision and Analysis

I decided to choose Option 2.

// Pseudo-code handling streaming from AI
const handleAsk = async (prompt) => {
  const response = await fetch('/api/ai/stream', { method: 'POST', body: prompt });
  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    const chunk = decoder.decode(value);
    setAiResponse(prev => prev + chunk); // Characters appear gradually on the UI
  }
};

Impact on Performance: The initial response speed (Time to First Token) dropped below 1 second. However, continuously updating the state (re-rendering) while streaming can lag the UI if you don't know how to optimize memo or use specialized libraries.

Impact on Maintainability: Code becomes more complex on both Backend and Frontend. Debugging a continuous data stream is harder than debugging a single request.

Impact on Team: Juniors were initially very fearful of the "Stream" concept. But after becoming familiar with ReadableStream, they expanded their thinking on how data moves across the Internet.

Self-Reflection: Was it Over-engineering?

I asked myself: Could a better Spinner have solved the problem? The answer is NO. In the ChatGPT era, streaming is no longer a "nice-to-have" feature; it is a mandatory standard for all AI applications. If you don't have it, your product looks like it's from 10 years ago.

If I were to go back, would I choose differently? Absolutely not. This decision completely changed how users received our product. Lesson learned: Sometimes, changing a fundamental architecture is the price you must pay to achieve a breakthrough user experience.

---

Notes on patience and data flows.